Systems and methods for using metadata to enhance data management operations

ABSTRACT

Systems and methods for managing electronic data are disclosed. Various data management operations can be performed based on a metabase formed from metadata. Such metadata can be identified from an index of data interactions generated by a journaling module, and obtained from their associated data objects stored in one or more storage devices. In various embodiments, such processing of the index and storing of the metadata can facilitate, for example, enhanced data management operations, enhanced data identification operations, enhanced storage operations, data classification for organizing and storing the metadata, cataloging of metadata for the stored metadata, and/or user interfaces for managing data. In various embodiments, the metabase can be configured in different ways. For example, the metabase can be stored separately from the data objects so as to allow obtaining of information about the data objects without accessing the data objects or a data structure used by a file system.

RELATED APPLICATIONS

This application claims the benefit of priority under 35 U.S.C. § 119(e)of U.S. Provisional Application No. 60/740,686, entitled “Systems andMethod for Classifying Information in a Storage Network,” filed Nov. 28,2005, and U.S. Provisional Application No. 60/752,203, entitled “Systemsand Methods for Classifying and Transferring Information in a StorageNetwork,” filed Dec. 19, 2005, each of which is hereby incorporatedherein by reference in its entirety.

The present disclosure relates to U.S. patent application Ser. No.______ [Attorney Docket COMMV.032A], titled “METABASE FOR FACILITATINGDATA CLASSIFICATION,” U.S. patent application Ser. No. ______ [AttorneyDocket COMMV.034A], titled “SYSTEMS AND METHODS FOR USING METADATA TOENHANCE DATA IDENTIFICATION OPERATIONS,” U.S. patent application Ser.No. ______ [Attorney Docket COMMV.035A], titled “SYSTEMS AND METHODS FORUSING METADATA TO ENHANCE STORAGE OPERATIONS,” U.S. patent applicationSer. No. ______ [Attorney Docket COMMV.036A], titled “DATACLASSIFICATION SYSTEMS AND METHODS FOR ORGANIZING A METABASE,” U.S.patent application Ser. No. ______ [Attorney Docket COMMV.037A], titled“SYSTEMS AND METHODS FOR CATALOGING METADATA FOR A METABASE,” and U.S.patent application Ser. No. [Attorney Docket COMMV.038A], titled “USERINTERFACES AND METHODS FOR MANAGING DATA IN A METABASE,” each filed oneven date herewith and each hereby incorporated by reference herein intheir entirety.

One or more embodiments of the present disclosure may also be used withsystems and methods disclosed in the following patents and pending U.S.patent applications, each of which is hereby incorporated herein byreference in its entirety:

U.S. patent application Ser. No.09/354,058, entitled “HierarchicalBackup and Retrieval System,” filed Jul. 15,1999;

U.S. Pat. No. 6,418,478, entitled “Pipelined High Speed Data TransferMechanism,” issued Jul. 9, 2002;

U.S. patent application Ser. No.09/610,738, entitled “Modular Backup andRetrieval System Used in Conjunction with a Storage Area Network,” filedJul. 6, 2000;

U.S. Pat. No. 6,542,972, entitled “Logical View and Access to PhysicalStorage in Modular Data and Storage Management System,” issued Apr. 1,2003;

U.S. Pat. No. 6,658,436, entitled “Logical View and Access to DataManage by a Modular Data and Storage Management System,” issued Dec. 2,2003;

U.S. patent application Ser. No. 10/658,095, entitled “Dynamic StorageDevice Pooling in a Computer System,” filed Sep. 9, 2003;

U.S. patent application Ser. No. 10/262,556, entitled “Method forManaging Snapshots Generated by an Operating System or OtherApplication,” filed Sep. 30, 2002;

U.S. patent application Ser. No. 10/818,749, entitled “System and Methodfor Dynamically Performing Storage Operations in a Computer Network,”filed Apr. 5, 2004;

U.S. patent application Ser. No. 10/877,831, entitled “HierarchicalSystem and Method for Performing Storage Operations in a ComputerNetwork,” filed Jun. 25, 2004;

U.S. patent application Ser. No. ______, entitled “System and Method forContainerized Data Storage and Tracking,” filed Dec. 19, 2005 [AttorneyDocket No. 4982/93];

U.S. patent application Ser. No. ______, entitled “Systems and Methodsfor Granular Resource Management in a Storage Network,” filed Dec. 19,2005 [Attorney Docket No. 4982/84];

U.S. patent application Ser. No. 11/313,224, entitled “Systems andMethods for Performing Multi-Path Storage Operations,” filed Dec. 19,2005;

U.S. patent application Ser. No. ______, entitled “Systems and Methodsfor Migrating Components in a Hierarchical Storage Network,” filed Dec.19, 2005 [Attorney Docket No. 4982/95];

U.S. patent application Ser. No. _______, entitled “Systems and Methodsfor Unified Reconstruction of Data in a Storage Network,” filed Dec. 19,2005 [Attorney Docket No. 4982/97];

U.S. patent application Ser. No. ______, entitled “Systems and Methodsfor Resynchronizing Storage Operations,” filed Dec. 19, 2005 [AttorneyDocket No. 4982/98]; and

U.S. patent application Ser. No. ______, entitled “Systems and Methodsfor Hierarchical Client Group Management,” filed Dec. 19, 2005 [AttorneyDocket No. 4982/102].

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialthat is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosures, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever.

BACKGROUND

1. Field

Embodiments of the present disclosure relate generally to performingoperations on electronic data in a computer network. More particularly,embodiments of the present disclosure relate to detecting datainteractions within a computer network and/or performing storage-relatedoperations according to one or more classification paradigms.

2. Description of the Related Art

Current storage management systems employ a number of different methodsto perform storage operations on electronic data. For example, data canbe stored in primary storage as a primary copy or in secondary storageas various types of secondary copies including, as a backup copy, asnapshot copy, a hierarchical storage management copy (“HSM”), anarchive copy, and other types of copies.

A primary copy of data is generally a production copy or other “live”version of the data which is used by a software application and isgenerally in the native format of that application. Such primary copydata is typically intended for short term retention (e.g., several hoursor days) before some or all of the data is stored as one or moresecondary copies, such as, for example, to prevent loss of data in theevent a problem occurred with the data stored in primary storage.

Secondary copies include point-in-time data and are typically intendedfor long-term retention (e.g., weeks, months or years) before some orall of the data is moved to other storage or is discarded. Secondarycopies may be indexed so users can browse and restore the data atanother point in time. After certain primary copy data is backed up, apointer or other location indicia such as a stub may be placed in theprimary copy to indicate the current location of that data.

One type of secondary copy is a backup copy. A backup copy is generallya point-in-time copy of the primary copy data stored in a backup format,as opposed to a native application format. For example, a backup copymay be stored in a backup format that facilitates compression and/orefficient long-term storage. Backup copies generally have relativelylong retention periods and may be stored on media with slower retrievaltimes than other types of secondary copies and media. In some cases,backup copies may be stored at on offsite location.

Another form of secondary copy is a snapshot copy. From an end-userviewpoint, a snapshot may be thought of as an instant image of theprimary copy data at a given point in time. A snapshot generallycaptures the directory structure of a primary copy volume at aparticular moment in time and may also preserve file attributes andcontents. In some embodiments, a snapshot may exist as a virtual filesystem, parallel to the actual file system. Users typically gainread-only access to the record of files and directories of the snapshot.By electing to restore primary copy data from a snapshot taken at agiven point in time, users may also return the current file system tothe state of the file system that existed when the snapshot was taken.

A snapshot may be created instantly, using a minimum amount of filespace, but may still function as a conventional file system backup. Asnapshot may not actually create another physical copy of all the data,but may simply create pointers that are able to map files anddirectories to specific disk blocks.

In some embodiments, once a snapshot has been taken, subsequent changesto the file system typically do not overwrite the blocks in use at thetime of the snapshot. Therefore, the initial snapshot may use only asmall amount of disk space needed to record a mapping or other datastructure representing or otherwise tracking the blocks that correspondto the current state of the file system. Additional disk space isusually required only when files and directories are actually modifiedlater. Furthermore, when files are modified, typically only the pointerswhich map to blocks are copied, not the blocks themselves. In someembodiments, for example in the case of copy-on-write snapshots, when ablock changes in primary storage, the block is copied to secondarystorage before the block is overwritten in primary storage. The snapshotmapping of file system data is also updated to reflect the changedblock(s) at that particular point in time.

An HSM copy is generally a copy of the primary copy data but typicallyincludes only a subset of the primary copy data that meets a certaincriteria and is usually stored in a format other than the nativeapplication format. For example, an HSM copy may include data from theprimary copy that is larger than a given size threshold or older than agiven age threshold and that is stored in a backup format. Often, HSMdata is removed from the primary copy, and a stub is stored in theprimary copy to indicate the new location of the HSM data. When a userrequests access to the HSM data that has been removed or migrated,systems use the stub to locate the data and often make recovery of thedata appear transparent, even though the HSM data may be stored at alocation different from the remaining primary copy data.

An archive copy is generally similar to an HSM copy. However, the datasatisfying criteria for removal from the primary copy is generallycompletely removed with no stub left in the primary copy to indicate thenew location (i.e., where the archive copy data has been moved to).Archive copies of data are generally stored in a backup format or othernon-native application format. In addition, archive copies are generallyretained for very long periods of time (e.g., years) and, in some cases,are never deleted. In certain embodiments, such archive copies may bemade and kept for extended periods in order to meet complianceregulations or for other permanent storage applications.

In some embodiments, application data over its lifetime moves from moreexpensive quick access storage to less expensive slower access storage.This process of moving data through these various tiers of storage issometimes referred to as information lifecycle management (“ILM”). Thisis the process by which data is “aged” from forms of primary storagewith faster access/restore times down through less expensive secondarystorage with slower access/restore times. For example, such aging mayoccur as data becomes less important or mission critical over time.

Regardless of where data is stored, conventional storage managementsystems perform storage operations associated with electronic data basedon location-specific criteria. For example, data generated byapplications running on a particular client is typically copiedaccording to location-specific criteria, such as from a specific folderor subfolder, according to a specified data path. A module installed onthe client or elsewhere in the system may supervise the transfer of datafrom the client to another location in a primary or secondary storage.

Similar data transfers associated with location-specific criteria areperformed when restoring data from secondary storage to primary storage.For example, to restore data a user or system process generally mustspecify a particular secondary storage device, piece of media, orarchive file. Thus, the precision with which conventional storagemanagement systems perform storage operations on electronic data isgenerally limited by the ability to define or specify storage operationsbased on data location.

Moreover, when identifying data objects, such as files associated withperforming storage operations, conventional storage systems often scanthe file system of a client or other computing device to determine whichdata objects on the client should be associated with the storageoperation. This may involve traversing the entire file system of theclient prior to performing storage operations. This process is typicallytime-consuming and uses significant client resources. In view of theforegoing, there is a need for systems and methods for performing moreprecise and efficient storage operations.

SUMMARY

Certain embodiments of the present disclosure provide systems andmethods for data classification to facilitate and improve datamanagement within an enterprise. In certain embodiments, the disclosedsystems and methods evaluate and define data management operations basedon data characteristics rather than, or in addition to, data location.Also provided are methods for generating a data structure of metadatathat describes system data and storage operations. In certainembodiments, this data structure may be consulted to determine changesin system data rather than scanning the data files themselves.

One embodiment of the present disclosure relates to a method formanaging electronic data in one or more storage devices. The methodincludes receiving from a journaling module an index of datainteractions made by one or more applications to one or more dataobjects on at least one storage device. The method further includesprocessing the index to identify at least one data interaction with atleast one of the data objects. The method further includes obtaining themetadata associated with the at least one data interaction. The methodfurther includes storing the metadata associated with the at least onedata interaction in a metabase.

In one embodiment, data object comprises a data file. In one embodiment,the metabase does not store the entire contents of the data objects. Inone embodiment, the metabase is stored separately from the data objects.In one embodiment, the metabase is stored on a storage device differentthan the at least one storage device. In one embodiment, the metabase isconfigured to allow obtaining of information about the plurality of dataobjects without accessing a data structure used by a file system. In oneembodiment, the data interactions are controlled by at least a firstprocessor and the metabase is controlled by a second processor. In oneembodiment, the method further includes accessing the metadata one ormore times to update the metabase. In one embodiment, the journalingmodule comprises an application separate from the applicationsassociated with the data interactions. In one embodiment, the methodfurther includes compiling the index of data interactions. In oneembodiment, the storing comprises updating the metadata stored in themetabase.

In one embodiment, the method additionally includes receiving aselection criteria, and accessing the metabase to identify which of theplurality of data objects satisfies the selection criteria withoutaccessing the plurality of data objects on the at least one storagedevice. In one embodiment, the selection criteria identifies dataobjects subject to a data migration policy.

In one embodiment, the method additionally includes initially populatingthe metabase by traversing the plurality of data objects to accessavailable metadata associated with the plurality of data objects.

Another embodiment of the present disclosure relates to a system formanaging electronic data in a network environment. The system includesat least one storage device configured to store one or more dataobjects. The system further includes a journal file populated by amonitoring module, with the journal file comprising a list of datainteractions between at least one application and at least of one of thedata objects. The system further includes a classification moduleconfigured to process the journal file to identify at least one datainteraction with at least one of the data objects, with theclassification module being further configured to obtain the metadataassociated with the data object. The system further includes a metabasecomprising a plurality of records, with at least one record storing themetadata associated with the data object.

In one embodiment, the at least one data object interaction comprises amodification of the at least one data object. In one embodiment, theclassification module is further configured to update the plurality ofrecords in the metabase based at least in part on the metadata obtainedfrom the data object. In one embodiment, for each listed datainteraction in the journal file, the journal file comprises at least oneof an update sequence number and a change journal reason code. In oneembodiment, the classification module is further configured to initiallypopulate the metabase by traversing the at least one storage device toaccess available metadata associated with the data objects. In oneembodiment, the system further includes a first processor and a secondprocessor. Operations associated with the monitoring module aresubstantially controlled by the first processor and operationsassociated with the metabase are substantially controlled by the secondprocessor. In one embodiment, the journal file is associated with afirst computing device and the metabase is associated with a secondcomputing device.

In one embodiment, the metabase comprises a centralized metabase incommunication with a plurality of computing devices. In one embodiment,each of the plurality of computing devices has its own journal file. Inone embodiment, at least some of the plurality of records includes anidentifier that indicates which computing device, and where on thatcomputing device, the metadata is located to allow comparison with aselection criteria without accessing the data objects.

In one embodiment, the classification module is located outside a dataflow path between the at least one application and the data objects onthe at least one storage device.

In one embodiment, the at least one storage device comprises a firststorage device and a second storage device. In one embodiment, thesystem further includes a management module configured to access themetabase to determine which of the plurality of data objects on thefirst storage device is subject to a data transfer policy. In oneembodiment, the management module performs the determination withoutaccessing the plurality of data objects on the first storage device. Inone embodiment, the first storage device comprises a primary storagedevice and the second storage device comprises a secondary storagedevice. In one embodiment, the management module is configured to copy afirst data object of the plurality of data objects on the first storagedevice to the second storage device. In one embodiment, the metabasecomprises first and second metabases, the first metabase having recordsstoring metadata associated with the data objects on the first storagedevice and the second metabase having records storing metadataassociated with the data objects on the second storage device. In oneembodiment, the metabase is updated according to locations of the dataobjects.

Yet another embodiment of the present disclosure relates to a system formanaging electronic data in a network environment. The system includesmeans for receiving from a journaling module an index of datainteractions between at least one application and one or more dataobjects. The system further includes means for processing the index toidentify at least one data interaction with at least one data object.The system further includes means for obtaining metadata associated withthe at least one data object. The system further includes means forstoring the metadata. The metadata in the means for storing is separatefrom the at least one data object.

In one embodiment, the means for processing is located outside a dataflow path between the at least one application and the plurality of dataobjects. In one embodiment, the journaling module is located in the dataflow path.

Yet another embodiment of the present disclosure relates to a method formanaging the transfer of electronic data in a storage system. The methodincludes receiving a monitor index having a plurality of entriesidentifying data interactions made by at least one application to aplurality of data objects on first and second storage devices, with theentries in the monitor index being populated by a journalingapplication. The method further includes updating, based on the monitorindex entries, metadata stored in a metabase. The method furtherincludes evaluating a selected criteria for transferring at least onedata object from the first storage device to the second storage device,with the evaluating comprising accessing the metadata stored in themetabase. The method further includes identifying a first data object,of the plurality of data objects, that satisfies the selected criteria.

In one embodiment, the method further includes performing at least onetransferring operation on the first data object based on the selectedcriteria. In one embodiment, the accessing of the metadata stored in themetabase is performed without accessing the data objects stored in thefirst storage device. In one embodiment, the method further includesaccessing one or more metadata associated with the corresponding dataobjects again one or more times to update the metabase. In oneembodiment, the identifying further comprises providing a file referencenumber associated with the first data object. In one embodiment, theidentifying further comprises providing a path indicative of a locationof the first data object on the first storage device. In one embodiment,the selected criteria comprises at least one of a schedule policy, astorage policy and a migration policy.

In one embodiment, the method additionally includes copying the firstdata object from a first location on the first storage device to asecond location on the second storage device. In one embodiment, themethod additionally includes updating the metabase to reflect the secondlocation of the first data object on the second storage device. In oneembodiment, the metabase comprises first and second metabases. The firstmetabase is associated with the first storage device and the secondmetabase is associated with the second storage device. In oneembodiment, the updating comprises updating one or more records of thesecond metabase. In one embodiment, the method additionally includesremoving records on the first metabase associated with the metadata ofthe first data object. In one embodiment, the method additionallyincludes storing on the first storage device reference data thatidentifies the second location of the first data object on the secondstorage device.

Yet another embodiment of the present disclosure relates to a system formanaging the transfer of electronic data between a plurality of storagedevices. The system includes a first storage device having a pluralityof data objects. The system further includes a second storage device.The system further includes a selected criteria for copying at least oneof the plurality of data objects on the first storage device to thesecond storage device. The system further includes at least one metabasehaving a plurality of records comprising metadata associated with theplurality of data objects. The system further includes a classificationmodule configured to evaluate a monitor index generated by a journalprogram. The monitor index includes a list of interactions between atleast one application and the plurality of data objects. Theclassification module is further configured to update the metadatastored in the at least one metabase based on interactions identified bythe monitor index. The system further includes a management moduleconfigured to access the at least one metabase to identify a first dataobject, of the plurality of data objects, that satisfies the selectedcriteria.

In one embodiment, the interactions identified by the monitor indexcomprise interactions that modify metadata of at least a portion of theplurality of data objects. In one embodiment, the management moduleaccesses the at least one metabase without accessing the plurality ofdata objects on the first storage device. In one embodiment, the journalprogram is different than the at least one application. In oneembodiment, the classification module is located outside an input/output(I/O) path between the at least one application and the first storagedevice. In one embodiment, the classification module is managed by afirst processor and the interactions are managed by a second processor.

In one embodiment, the management module is further configured totransfer the first data object from the first storage device to thesecond storage device. In one embodiment, the first storage devicecomprises a faster access time than the second storage device.

In one embodiment, the at least one metabase comprises a first metabaseand a second metabase. Records stored in the first metabase correspondto metadata of the plurality of data objects stored on the first storagedevice, and records stored in the second metabase correspond to metadataof a second plurality of data objects stored on the second storagedevice. In one embodiment, the system further includes a master indexidentifying information stored on the first and second metabases.

In one embodiment, at least a portion of the at least one metabase isstored on the first storage device.

Yet another embodiment of the present disclosure relates to a system formanaging the transfer of electronic data stored on a plurality ofstorage devices. The system includes first means for storing a firstplurality of data objects. The system further includes second means forstoring a second plurality of data objects. The system further includesmeans for identifying at least one of the first plurality of dataobjects to be transferred from the first storing means to the secondstoring means. The system further includes third means for storingmetadata associated with the first plurality of data objects. The systemfurther includes means for processing a journal file comprising a listof interactions between at least one application and the plurality ofdata objects, with the processing means being further configured toupdate the third storing means based on interactions identified by thejournal file. The system further includes means for accessing the thirdstoring means to identify a first data object, of the plurality of dataobjects, that satisfies the selected criteria.

In one embodiment, the metadata is stored separate from the entirecontents of the data objects. In one embodiment, the identifiedinteractions comprise interactions that modify metadata of at least aportion of the plurality of data objects. In one embodiment, theaccessing of the third storing means is performed without accessing theplurality of data objects on the first storing means. In one embodiment,the selected criteria comprises a data storage migration policy.

Yet another embodiment of the present disclosure relates to a method forperforming data identification operations involving electronic data. Themethod includes receiving an index generated by a journal application,with the index identifying data interactions made by at least one otherapplication to one or more data objects on a storage device. The methodfurther includes updating, based on the index, metadata stored in ametabase. The method further includes receiving selected criteriaidentifying one or more properties of at least one data object. Themethod further includes accessing the metabase according to the selectedcriteria without accessing the data objects on the storage device. Themethod further includes identifying, based on the accessing, the atleast one data object that satisfies the selected criteria.

In one embodiment, the metadata in the metabase are stored separate fromthe entire content of each of the data objects. In one embodiment,accessing of the metabase is performed without accessing the dataobjects on the storage device. In one embodiment, the method furtherincludes accessing one or more of the metadata associated with thecorresponding data objects again one or more times to update themetabase.

In one embodiment, the method additionally includes receiving theselected criteria from a user. In one embodiment, the methodadditionally includes providing a graphical user interface for promptingthe selected criteria.

In one embodiment, the selected criteria comprises identification andtracking of the data interactions on a user or group basis. In oneembodiment, the data interactions comprise at least one of a readoperation, a write operation and an access operation. In one embodiment,the access operation comprises at least one of web-page access,electronic gaming interaction, chat-room access, instant messaging,communication interaction and multimedia interaction.

In one embodiment, the accessing comprises accessing a plurality ofmetabases. In one embodiment, the at least one storage device comprisesa plurality of storage devices and each of the plurality of metabases isassociated with one of the plurality of storage devices. In oneembodiment, the method additionally includes accessing a metadata indexto determine which of the plurality of metabases is likely to containinformation relating to the selected criteria.

In one embodiment, the identifying comprises providing a storage devicepath of the at least one data object. In one embodiment, the identifyingcomprises providing a file reference number of the at least one dataobject.

Yet another embodiment of the present disclosure relates to a system forperforming data identification operations involving electronic data. Thesystem includes at least one storage device configured to store aplurality of data objects having metadata and content associatedtherewith. The system further includes a journal file maintained by amonitoring module, with the journal file identifying data operationsdirected to one or more of the plurality of data objects by at least oneapplication. The system further includes at least one metabasecomprising a plurality of records storing the metadata associated withthe plurality of data objects, with the plurality of records beingupdated based on information obtained from the journal file. The systemfurther includes a processor configured to receive a selected criteriaand to access the at least one metabase to identify at least one dataobject of the plurality of data objects that satisfies the selectedcriteria.

In one embodiment, the at least one metabase does not store the entirecontent of each of the plurality of data objects. In one embodiment, theaccess of the at least one metabase is performed without accessing theplurality of data objects on the at least one storage device. In oneembodiment, the system further includes a classification moduleconfigured to update the plurality of records on the at least onemetabase, with the classification module being located outside a dataflow path between the at least one application and the plurality of dataobjects.

In one embodiment, the processor comprises first and second processors.The first processor is configured to access the at least one metabaseand the second processor is configured handle the data operationsassociated with the plurality of data objects on the at least onestorage device. In one embodiment, the first processor is associatedwith a first computing device and the second processor is associatedwith a second computing device.

In one embodiment, the at least one storage device comprises multiplestorage devices and the at least one metabase comprises multiplemetabases, with each of the multiple metabases being associated with oneof the multiple storage devices. In one embodiment, the multiple storagedevices comprise a primary storage device having production data and asecondary storage device having one or more copies of at least some ofthe production data. In one embodiment, the secondary storage device hasat least some additional data other than the one or more copies of theat least some of the production data. In one embodiment, the processoris further configured to access each of the multiple metabases.

Yet another embodiment of the present disclosure relates to a system forperforming data identification operations involving electronic data. Thesystem includes means for storing a plurality of data objects. Thesystem further includes means for identifying data operations directedto one or more of the data objects by at least one application, with themeans for identifying being generated by a journaling module. The systemfurther includes means for storing metadata associated with the dataobjects, where the means for storing metadata is updated based onselected ones of the data operations identified by the means foridentifying. The system further includes means for accessing the meansfor storing metadata based on at least one selected criteria to identifyat least one data object corresponding to the selected criteria withoutaccessing the data objects located on the means for storing the dataobjects.

In one embodiment, the selected ones of the data operations result inmodification of the metadata of at least one of the data objects. In oneembodiment, the identifying of the at least one data object is performedwithout accessing the data objects on the first storing means. In oneembodiment, the system further includes means for receiving the selectedcriteria.

Yet another embodiment of the present disclosure relates to a method formanaging electronic data in a storage system. The method includesstoring a plurality of data objects in a storage device. The methodfurther includes receiving a log having a plurality of entriesidentifying data interactions between at least one application and oneor more of the plurality of data objects, with the log entries beingpopulated by a journaling application. The method further includesupdating, based on the log entries, metadata stored in a metabase. Themethod further includes accessing a selection criteria regardingperforming at least one storage operation on at least one of theplurality of data objects based on the metadata. The method furtherincludes identifying at least a first data object of the plurality ofdata objects based on the metadata that satisfies the selectioncriteria.

In one embodiment, the metadata in the metabase are stored separate fromentire content of each of the plurality of data objects. In oneembodiment, the identifying is performed without accessing the pluralityof data objects on the storage device.

In one embodiment, the updating comprises processing the log to selectat least one data interaction that modifies the metadata of at least oneof the plurality of data objects, and updating the metadata in themetabase based on the selected at least one data interaction. In oneembodiment, the identifying is performed without accessing a datastructure associated with storage of the plurality of data objects onthe storage device. In one embodiment, the identifying is performedwithout accessing a file access table (FAT) associated with storage ofthe plurality of data objects on the storage device.

In one embodiment, the method additionally includes performing the atleast one storage operation on the first data object. In one embodiment,the at least one storage operation comprises a copy operation. In oneembodiment, the at least one storage operation comprises a snapshotoperation. In one embodiment, the at least one storage operationcomprises a hierarchical storage management (HSM) operation. In oneembodiment, performing the at least one storage operation comprisescopying the first data object to a second storage device. In oneembodiment, the method additionally includes updating records of asecond metabase associated with the second storage device to include themetadata of the first data object.

In one embodiment, the selection criteria comprises a user-definedpolicy. In one embodiment, the selection criteria identifies metadatarelating to at least one of a read operation, a write operation, and anaccess operation associated with the data object. In one embodiment, theaccess operation comprises at least one of web-page access, electronicgaming interaction, chat-room access, instant messaging, communicationinteraction, or multimedia interaction. In one embodiment, the selectioncriteria involving the access operation allows at least one ofidentifying, capturing, classifying, or tracking of the accessoperation.

In one embodiment, the method further includes accessing one or more ofthe metadata associated with the corresponding data objects one or moretimes to update the metabase.

Yet another embodiment of the present disclosure relates to a system formanaging electronic data stored on a storage device. The system includesa selection criteria for performing a storage operation on at least oneof a plurality of data objects on a storage device, with the selectioncriteria being dependent upon metadata, other than location data,associated with the plurality of data objects. The system furtherincludes a metabase having a plurality of records storing the metadataassociated with the plurality of data objects. The system furtherincludes a classification module configured to process a log filegenerated by a journal program. The log file includes a list ofinteractions between at least one application and the plurality of dataobjects. The classification module is further configured to update themetabase based on interactions identified in the log file. The systemfurther includes a management module configured to access the metabaseto identify at least a first data object that satisfies the selectioncriteria.

In one embodiment, the metadata in the metabase are stored separate fromentire content of the plurality of data objects. In one embodiment, theclassification module is configured to update the metabase based oninteractions that modify metadata of at least a portion of the pluralityof data objects. In one embodiment, the management module accesses themetabase to identify the first data object without accessing either theplurality of data objects or a data structure used in storage of theplurality of data objects on the storage device.

In one embodiment, the management module is configured to copy the firstdata object from the storage device to a second storage device. In oneembodiment, the second storage device comprises a storage deviceconfigured to retain the copy of the first data object. In oneembodiment, the plurality of records on the metabase store metadataassociated with data objects on the second storage device. In oneembodiment, the system further includes a second metabase having asecond plurality of records storing metadata associated with dataobjects on the second storage device. In one embodiment, the selectioncriteria comprises a data migration policy.

Yet another embodiment of the present disclosure relates to a system forperforming storage operations on electronic data. The system includesmeans for storing a plurality of data objects. The system furtherincludes means for identifying at least one storage operation to beperformed on one or more data objects based on the metadata associatedwith the plurality of data objects. The system further includes meansfor storing the metadata. The system further includes means forprocessing a journal file comprising a list of interactions between atleast one application and one or more of the plurality of data objects,with the processing means being further configured to update the meansfor storing the metadata based on interactions identified by the journalfile. The system further includes means for performing a storageoperation on the at least one data object, based on accessing themetadata in the means for storing the metadata.

In one embodiment, the interactions identified by the journal filecomprised interactions that modify metadata of at least a portion of theplurality of data objects. In one embodiment, accessing of the metadatain the means for storing the metadata is performed without accessing theplurality of data objects on the means for storing the data objects. Inone embodiment, the at least one application is managed by a firstprocessor and the processing means is managed by a second processor.

Yet another embodiment of the present disclosure relates to a method ofclassifying data in a storage system. The method includes processing anindex of data interactions between at least one application and one ormore data objects on at least one storage device, with the index beingpopulated by a journaling module. The method further includes selectingan entry from the index corresponding to a first data interactioninvolving a first data object. The method further includes obtainingfrom the selected entry information indicative of metadata associatedwith the first data object. The method further includes updating ametabase with the information obtained from the selected entry, with themetabase having a plurality of records storing metadata associated withthe data objects.

In one embodiment, the selected entry information comprises informationindicative of modifications to metadata associated with the first dataobject. In one embodiment, the metadata in the metabase is storedseparately from the entire contents of the data objects. In oneembodiment, the method further includes accessing one or more of themetadata associated with the corresponding data objects again one ormore times to update the metabase.

In one embodiment, the updating comprises determining whether theselected entry in the index of data interactions has an existing recordin the metabase; if no record exists corresponding to the selectedentry, creating a new record in the metabase; and updating the existingrecord or the new record with at least a part of the informationobtained from the selected entry.

In one embodiment, the selecting comprises determining whether the entryis a new entry in the index of data interactions. In one embodiment, theentry is considered to be new if a time stamp of the entry is later thana time at which a previous entry was analyzed. In one embodiment, theentry is considered to be new based on an identifier of the entry. Inone embodiment, the identifier comprises an update sequence number thatidentifies the entry in the index of data interactions.

In one embodiment, the method further includes initially populating themetabase by accessing the data objects so as to access availablemetadata associated with the data objects. In one embodiment, the methodadditionally includes quiescing the data interactions associated withthe at least one storage device, and performing the populating duringthe quiescing. In one embodiment, the populating is performed duringoperation of the at least one storage device. In one embodiment, themethod additionally includes queuing the data interactions generatedduring the populating to allow capture of the data interactions duringthe accessing process.

In one embodiment, the method additionally includes receiving inputregarding a user-defined tag. The obtaining information is based atleast in part on the user-defined tag. In one embodiment, the updatingfurther comprises determining which of a plurality of metabasescomprises records storing metadata associated with the first dataobject.

Yet another embodiment of the present disclosure relates to a system formanaging electronic data in a storage network. The system includes adata classification module configured to scan entries in a journal fileof data interactions between at least one application and one or moredata objects stored on at least one storage device. The dataclassification module is further configured to select an entry from theindex corresponding to a first data interaction involving a first dataobject. The data classification module is further configured to classifyone or more properties of the data object based on information obtainedfrom the selected entry. The data classification module is furtherconfigured to update the one or more classified properties of the dataobject in a metabase.

In one embodiment, the journal file is populated by a monitoring module.In one embodiment, the data classification module is further configuredto access the one or more data objects one or more times to update themetabase. In one embodiment, the properties of the data objects arestored in the metabase separately from entire content of the dataobjects.

In one embodiment, the information obtained from the selected entry isindicative of modifications to metadata of the first data objectresulting from the first data interaction. In one embodiment, themetadata comprises at least one of: a data owner, a last modified time,a last accessed time, a data object size and an application type.

In one embodiment, the data classification module is further configuredto classify the one or more properties of the data object based onuser-defined tags.

In one embodiment, the data classification module is further configuredto periodically scan the entries in the index. In one embodiment, thedata classification module is further configured to allow analysis ofthe one or more properties of the data objects based on a selectedcriteria without accessing the data objects.

Yet another embodiment of the present disclosure relates to a system formanaging electronic data. The system includes means for storing a listof data object interactions between at least one application and one ormore data objects. The system further includes means for storingmetadata associated with the one or more data objects. The systemfurther includes means for identifying from the list of data objectinteractions a first data object interaction involving metadatamodifications to a first data object, with the identifying means beingfurther configured to update a record in the means for storing metadataassociated with the metadata of the first data object.

In one embodiment, the means for storing is generated by a journalingapplication. In one embodiment, the means for storing metadata comprisesmeans for storing the metadata separate from the entire content of eachof the data objects. In one embodiment, the means for identifying isfurther configured to initially populate the means for storing metadataby accessing the data objects so as to access available metadataassociated with the data objects. In one embodiment, the means foridentifying is configured to process the list of data objectinteractions by accessing an identifier of each entry in the list.

Yet another embodiment of the present disclosure relates to a method formanaging electronic data in a data storage network. The method includesreceiving a classification criteria that identifies at least oneproperty associated with one or more data objects. The method furtherincludes monitoring a plurality of data interactions involving at leastone application and the one or more data objects. The method furtherincludes determining, from the data interactions, select datainteractions that satisfy the classification criteria, with each selectdata interaction being indicative of changes to the propertiesassociated with the one or more data objects. The method furtherincludes maintaining a metabase based at least in part on the selectdata interactions, with the metabase storing at least the properties ofthe one or more data objects.

In one embodiment, the classification criteria comprises a user-definedrule. In one embodiment, the one or more properties comprise at leastone of a file owner and a permission level. In one embodiment, themetabase stores a path of one or more of the data objects correspondingto the select data interactions. In one embodiment, the metabase storesa reference number of the data object corresponding to one of the datainteractions. In one embodiment, the maintaining comprises periodicallyprocessing and updating the metabase based on modifications associatedwith the select data interactions.

In one embodiment, the method further includes recording the datainteractions in a journal index. In one embodiment, the recordingfurther comprises recording metadata corresponding to the select datainteractions.

In one embodiment, the metabase stores the properties of the dataobjects without storing the entire contents of the data objects.

Yet another embodiment of the present disclosure relates to a system forcataloging electronic data associated with data objects stored on atleast one storage device. The system includes a journaling moduleconfigured to receive a classification criteria. The journaling moduleis further configured to track a plurality of data interactionsinvolving at least one application and one or more data objects storedon at least one storage device. The journaling module is furtherconfigured to identify, from the plurality of data interactions, selectdata interactions that satisfy the classification criteria. The systemfurther includes a classification module configured to maintain a datastructure of metadata associated with the select data interactions withthe one or more data objects.

In one embodiment, each of the select data interactions result in achange to metadata of a corresponding one of the data objects. In oneembodiment, the journal module is further configured to record theselect data interactions in a log.

In one embodiment, the metabase comprises a plurality of metabases andthe at least one storage device comprises a plurality of storagedevices. In one embodiment, each of the plurality of metabases isassociated with one of the plurality of storage devices. In oneembodiment, the entry is recorded in one or more of a plurality of logs,with each log being associated with one of the plurality of metabases.

In one embodiment, the system further includes a second storage devicefor storing the log. In one embodiment, the system further includes auser interface module configured to receive the classification criteriafrom a user. In one embodiment, the journaling module is furtherconfigured to tag certain entries based on user-defined data objectproperties.

In one embodiment, the system further includes a processor configured tocompare a selection criteria and the metadata in the metabase withoutaccessing the plurality of data objects on the at least one storagedevice. In one embodiment, the selection criteria is associated withidentifying which of the plurality of data objects satisfies a datamigration policy.

Yet another embodiment of the present disclosure relates to a system forcataloging electronic data in a data storage network. The systemincludes means for receiving classification criteria. The system furtherincludes means for monitoring a plurality of data interactions betweenat least one application and one or more data objects stored on at leastone storage device. The monitoring means is further configured todetermine, from the plurality of data interactions, select datainteractions that satisfy the classification criteria. The monitoringmeans is further configured to record an entry for each select dataobject interaction. The system further includes means for maintaining adata structure of metadata associated with the select data interactionswith the one or more data objects.

In one embodiment, each of the select data object interactions isindicative of a modification to metadata of a corresponding one of theof data objects. In one embodiment, the system further includes meansfor processing a journal file and updating the metabase based onmodifications to metadata resulting from select data interactionsidentified in the journal file. In one embodiment, the system furtherincludes means for performing data migration of a portion of theplurality of data objects based on the data structure of metadata. Inone embodiment, the system further includes means for comparing aselection criteria with the metabase without accessing the data objectson the at least one storage device to identify the portion of dataobjects that relate to the selection criteria.

Yet another embodiment of the present disclosure relates to a system formanaging electronic data in a network environment. The system includesan index generated by a journaling module, with the index comprising alist of one or more data interactions with one or more data objects. Thesystem further includes an interface configured to receive at least oneselection criteria regarding at least one attribute of the one or moreof the data objects. The system further includes metadata stored in adata structure where the metadata is associated with the one or moredata objects. The system further includes a classification moduleconfigured to process the index to identify data interactions thatsatisfy the selection criteria, with the classification module beingfurther configured to update the metadata in the data structure based onthe identified data object interactions.

In one embodiment, the list of data interactions comprises a list ofinteractions between a memory and the data objects on a storage device.In one embodiment, the interface comprises a user interface. In oneembodiment, the selection criteria comprises criteria for migrating oneor more data objects from a first storage device to a second storagedevice. In one embodiment, the data structure that stores the metadatais stored separately from the entire contents of the data objects.

In one embodiment, the selection criteria comprises a policy forperforming a storage operation on at least one of the plurality of dataobjects. In one embodiment, the policy identifies a plurality of datafile attributes.

In one embodiment, the at least one attribute comprises a user- definedattribute. In one embodiment, the interface is configured to provide auser with an option to select at least one of a plurality of predefinedattributes.

Yet another embodiment of the present disclosure relates to a method formanaging electronic data in a storage system. The method includesreceiving from a journaling module a list of data interactions made byat least one application to at least one or more data objects on atleast one storage device, with each of the data objects having metadataand content associated therewith. The method further includesmaintaining metadata associated with the data objects. The maintainingfurther includes processing the data interactions made to one or more ofthe data objects, obtaining the metadata of the one or more dataobjects, and storing the metadata in a data structure. The methodfurther includes allowing a user interaction to obtain information fromthe metadata about one or more properties of the data objects withoutaccessing the data objects on the storage device.

In one embodiment, the at least one data interaction results in amodification of metadata of at least one of the data objects. In oneembodiment, the user interaction comprises a selection criteria input bya user.

In one embodiment, the user interaction comprises performing a storagemigration operation. In one embodiment, the user-defined interactioncomprises performing a scheduling operation.

Yet another embodiment of the present disclosure relates to a system formanaging electronic data in a storage system. The system includes meansfor providing a list of data interactions, with the providing meansbeing generated by a journal application. The system further includesmeans for receiving from a user a selection criteria regarding at leastone attribute of one or more data objects. The system further includesmeans for storing metadata associated with the data objects. The systemfurther includes means for processing the data interactions to identifydata object interactions that satisfy the selection criteria. Theprocessing means is further configured to update the metadata in themeans for storing based on the identified data interactions.

In one embodiment, the list of data interactions comprises a list ofinteractions between a memory and data objects stored on a storagedevice. In one embodiment, the storing means comprises storing metabaseassociated with the data objects separate from the entire content of thedata objects. In one embodiment, the selection criteria comprisesmigrating one or more data objects on a first storage device to a secondstorage device. In one embodiment, the selection criteria comprisesperforming a storage operation. In one embodiment, the selectioncriteria comprises a data migration policy. In one embodiment, the meansfor receiving comprises a graphical user interface.

Yet another embodiment of the present disclosure relates to a system formanaging electronic data in a network environment. The system includes adata structure comprising metadata associated with a plurality of dataobjects, with the metadata resulting from processing a list of datainteractions generated by a journaling module. The system furtherincludes an interface configured to receive a selection criteriaregarding at least one attribute of one or more of the data objects. Thesystem further includes a classification module configured to processthe metadata to identify one or more data interactions that satisfy theselection criteria.

In one embodiment, the classification module is further configured toprovide information about the one or more identified data interactionsto the interface.

For purposes of summarizing the disclosure, certain aspects, advantagesand novel features of the present disclosure have been described herein.It is to be understood that not necessarily all such advantages may beachieved in accordance with any particular embodiment of the presentdisclosure. Thus, the present disclosure may be embodied or carried outin a manner that achieves or optimizes one advantage or group ofadvantages as taught herein without necessarily achieving otheradvantages as may be taught or suggested herein.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present disclosure are illustrated in the figures ofthe accompanying drawings, which are meant to be exemplary and notlimiting, and in which like references are intended to refer to like orcorresponding parts.

FIG. 1 illustrates a flow chart of a metadata storage process inaccordance with embodiments of the present disclosure.

FIG. 2 illustrates a block diagram of an exemplary embodiment of a dataclassification system.

FIG. 3A illustrates a block diagram of another exemplary embodiment of adata classification system.

FIG. 3B illustrates a flow chart of a journaling process in accordancewith embodiments of the present disclosure.

FIG. 4 illustrates a flow chart of a data classification process inaccordance with embodiments of the present disclosure.

FIG. 5 illustrates a block diagram of another exemplary embodiment of adata classification.

FIG. 6 illustrates a flow chart of a metabase query process inaccordance with embodiments of the present disclosure.

FIG. 7 illustrates a block diagram of another embodiment of a dataclassification system having a centralized metabase

FIG. 8 illustrates a flow chart of a data classification process usableby the data classification system of FIG. 7, in accordance withembodiments of the present disclosure.

FIG. 9 illustrates a block diagram of an exemplary embodiment of a dataclassification system having a network attached storage (NAS) device.

FIG. 10 illustrates a flow chart of a data classification process usableby the data classification system of FIG. 9, in accordance withembodiments of the present disclosure.

FIG. 11 illustrates a flow chart of an exemplary embodiment of a searchprocess for a multiple metabase system.

FIG. 11A illustrates another embodiment of a data classification systemusable with a variety of applications.

FIG. 12 illustrates a flow chart of a data classification process usableby the data classification system of FIG. 11A, in accordance withembodiments of the present disclosure.

FIG. 13 illustrates a block diagram of an exemplary embodiment of asystem for communicating metadata between multiple computing devices.

FIG. 14 illustrates a flow chart of a metadata transfer process usableby the system of FIG. 13 in accordance with embodiments of the presentdisclosure.

FIG. 15 illustrates a block diagram of an exemplary embodiment of anetwork system for performing data classification.

FIG. 15A illustrates a block diagram of an exemplary embodiment of anetwork system.

FIG. 15B illustrates a block diagram of another exemplary embodiment ofa network system.

FIG. 16 illustrates an exemplary embodiment of a data structure for anarchive file.

FIG. 17 illustrates a flow chart of a data transfer process, inaccordance with embodiments of the present disclosure.

FIG. 18 illustrates an exemplary embodiment of a storage operation cellhierarchy, in accordance with embodiments of the present disclosure.

FIG. 19 illustrates a flow chart of a search process usable acrossmultiple primary and/or secondary storage devices, in accordance withembodiments of the present disclosure.

FIG. 20 illustrates a flow chart of a data retrieval process, inaccordance with embodiments of the present disclosure.

DETAILED DESCRIPTION OF SOME EMBODIMENTS

The features of the systems and methods will now be described withreference to the drawings summarized above. Throughout the drawings,reference numbers are re-used to indicate correspondence betweenreferenced elements. The drawings, associated descriptions, and specificimplementation are provided to illustrate embodiments of the presentdisclosure and not to limit the scope of the disclosure.

Embodiments of the present disclosure are generally concerned withsystems and methods that analyze and classify various forms of datathat, among other things, facilitate identification, searching, storageand/or retrieval of data that satisfies certain criteria. Althoughdescribed in connection with certain specific embodiments, it will beunderstood that various features disclosed herein can have broad-basedapplicability to any wireless or hard-wired network or data transfersystem that stores and/or conveys data from one point to another,including communication networks, enterprise networks, combinations ofthe same storage networks, and the like.

Generally, certain embodiments of the present disclosure include systemsand methods for analyzing data and other electronic information in acomputer network (sometimes referred to herein as a “data objects”). Forinstance, certain embodiments concern creating a database or index ofinformation that describes certain pertinent aspects of the data objectsthat allow a user or system process to consult the database to obtaininformation regarding the network data. For example, a data collectionagent may traverse a network file system and obtain certaincharacteristics and other attributes of data in the system. In someembodiments, such a database may be a collection of metadata and/orother information regarding the network data and is referred to hereinas a “metabase.” Generally, metadata refers to data or information aboutdata, and may include, for example, data relating to storage operationsor storage management, such as data locations, storage managementcomponents associated with data, storage devices used in performingstorage operations, index data, data application type, combinations ofthe same or other data.

In certain embodiments, if it is desired to obtain information regardingnetwork data, a system administrator or system process may consult themetabase for such information rather than iteratively access and analyzeeach data item in the network. Accessing the metabase may significantlyreduce the amount of time required to obtain data object information byreducing the need to obtain information from the source data.Furthermore, such embodiments may reduce the involvement of networkresources in this process, thereby reducing the processing burden on thehost system.

Examples of various types of data and operations on such data arefurther described in the above-referenced related applications. Oneexample of a system that performs storage operations on electronic datais the QiNetix® storage management system by CommVault Systems ofOceanport, New Jersey.

The QiNetix® system leverages a modular storage management architecturethat may include, among other things, storage manager components, clientor data agent components, and/or media agent components. Additionaldetails of the QiNetix® system are further described in U.S. patentapplication Ser. No. 10/818,749, filed Apr. 5, 2004, which is herebyincorporated herein by reference in its entirety. The QiNetix® systemalso may be hierarchically configured into backup cells to store andretrieve backup copies of electronic data as further described in U.S.patent application Ser. No.09/354,058.

FIG. 1 illustrates flow chart of a metadata storage process 100according to certain embodiments of the present disclosure. In order toperform the some of the functions described herein, it may be necessaryat the outset to install certain data classification software or dataclassification agents on computing devices within the network (block102). This may be done, for example, by installing classificationsoftware on client computers and/or servers within a given network. Insome embodiments, classification agents may be installed globally on acomputing device or with respect to certain subsystems on a computingdevice. The classification software may monitor certain informationregarding data objects generated by the computers and classify thisinformation for use as further described herein.

Next, at block 104, a monitor agent is initialized. Such a monitoringagent may be resident or installed on each computing device similar tothe deployment of classification agents described above and may beconfigured to monitor and record certain data interactions within eachmachine or network process. For example, the monitor agent may include afilter driver program and may be deployed on an input/output port ordata stack and operate in conjunction with a file management program torecord interactions with computing device data. Such operation mayinvolve creating a data structure such as a record or journal of eachinteraction. The records may be stored in a journal data structure andmay chronicle data interactions on an interaction by interaction basis.The journal data structure may include information regarding the type ofinteraction that has occurred along with certain relevant properties ofthe data involved in the interaction. One example of such a monitorprogram may include Microsoft's Change Journal.

Prior to populating a metabase with metadata, in certain embodiments,portions of the network or subject system may be quiesced such that datainteractions are not permitted prior to completing an optional scan ofsystem files as described in conjunction with block 106 below. This maybe done in order to obtain an accurate picture of the data being scannedand/or to maintain referential integrity within the system. For example,if the system was not quiesced, data interactions would continue and beallowed to change data in the mass storage. However, in someembodiments, the subject system may be allowed to continue to operate,with operations or instructions queued in a cache. In such embodiments,these operations are performed after the scan is complete so that anydata interactions that occur based on the cached operations are capturedby the monitor agent.

In certain embodiments, the file scanning of block 106 is performed by adata classification agent or module. Such scanning may includetraversing the file system of a client to identify data objects or otherfiles, email or other information currently stored or present in thesystem and/or to obtain certain information, such as metadata, regardingthe information. In certain embodiments metadata may include informationabout data objects or characteristics associated with the data objects,such as one or more of the following, but not limited to: the data owner(e.g., the client or user that generates the data or other datamanager), the last modified time (e.g., the time of the most recentmodification of the data object), a data object size (e.g., number ofbytes of data), information about the data content (e.g., applicationthat generated the data and/or user that generated the data), to/frominformation for email (e.g., an email sender, recipient, individual orgroup on an email distribution list), creation date (e.g., date on whichthe data object was created), file type (e.g., format or applicationtype), last accessed time (e.g., time the data object was most recentlyaccessed or viewed), application type (e.g., application that generatedthe data object), location/network (e.g., a current, past or futurelocation of the data object and network pathways to/from the dataobject), frequency of change (e.g., a period in which the data object ismodified), business unit (e.g., a group or department that generates,manages or is otherwise associated with the data object), and aginginformation (e.g., a schedule, such as a time period, in which the dataobject is migrated to secondary or long term storage), combinations ofthe same or the like. In certain embodiments, the information obtainedin the scanning process may be advantageously used to initially populatethe metabase of information regarding network data at block 108.

After the metabase has been populated, the network or subject system maybe released from its quiesced state. Next, at block 110, the monitoragent monitors system operations to record changes to system data, suchas in the change journal database as described above. In certainembodiments, the change journal database includes a database of metadataor data changes and may comprise, for example, log files of the data ormetadata changes. In some embodiments, the data classification agent mayperiodically consult the change journal database for new entries. If newentries exist, the data classification agent may examine the entries,and if deemed relevant, the entries may be analyzed, parsed, and writtento the metabase as an update (block 112). In other embodiments, changejournal entries may be supplied substantially in parallel to the journaldatabase and data classification agent. This allows the metabase tomaintain substantially current information regarding the state of systemdata at any given point in time.

As mentioned above, one benefit of such a metabase is that the metabase,in certain embodiments, significantly reduces the amount of timerequired to obtain information by substantially eliminating the need toobtain information directly from the source data. For example, assume asystem administrator desires to identify data objects that a certainuser has interacted with that contain certain content or othercharacteristics. Rather than search each file in each directory, whichcan be a very time consuming process, especially when the data objectsreside on multiple storage devices, the administrator may search themetabase to identify such data objects and properties (e.g., metadata)associated with those objects.

Moreover, in certain embodiments, use of the metabase for satisfyingdata queries may also reduce the involvement of network resources inthis process, substantially reducing the processing burden on the hostsystem. For example, as described above, if an administrator desires toidentify certain data objects, querying the metabase rather than thefile system virtually removes the host system from the query process(e.g., no brute force scanning of directories and files is required),allowing the host system to continue performing host tasks rather thanbe occupied with search tasks.

FIG. 2 depicts a client 200 constructed in accordance with certainembodiments of the present disclosure. As shown, the client 200 includesa classification agent 202 and a monitor agent 204, combined as anupdate agent 206, which may be a single module encompassing thefunctionality of both agents. In yet other embodiments, theclassification agent 202 and monitor agent 204 may be two separatemodules configured to communicate with each other. Client 200 alsoincludes data store 210, a metabase 212, and a change record module 214.In certain embodiments, one or more of the data store 210, the metabase212 and the changed record module 214 are external and/or internal tothe client 200.

Generally, the client 200 may be a computing device, or any portion of acomputing device that generates electronic data. Data store 208generally represents application data such as production volume dataused by client 200. Metabase 212, which may be internal or external toclient 200 may contain information generated by classification agent 202as described above. Similarly, changed record journal 214, which alsomay be internal or external to client 200 may contain informationgenerated by monitor agent 204 as described above.

In operation, data interactions occurring within client 200 may bemonitored with update agent 206 or monitor agent 204. Any relevantinteraction may be recorded and written to change record 214. Dataclassification agent 202 may scan or receive entries from monitor agent204 and update metabase 212 accordingly. Moreover, in the case whereupdate agent 206 is present, monitored data interactions may beprocessed in parallel with updates to change record 214 and written todata store 210 and metabase 212 occurring accordingly. A file system 207may be used to conduct or process data from the client to a data store210.

FIG. 3A provides a more detailed view of the journaling andclassification mechanisms of client 200 generally shown in FIG. 2. Asshown, system 300 may include a memory 302, an update agent 304 whichmay include a separate or integrated monitor agent 306, a classificationagent 312 a and/or 312 b, and a content agent 315, a monitor programindex 310, metabase 314 and mass storage device 318.

In operation, data interactions that occur between memory 302 and massstorage device 318 may be monitored by monitor agent 306. In someembodiments, memory 302 may include random access memory (RAM) or othermemory device employed by a computer in client 200 in performing dataprocessing tasks. Certain information from memory 302 may beperiodically read or written to mass storage device 318 which mayinclude a magnetic or optical disk drive such as a hard drive or otherstorage device known in the art. Such data interactions are monitored bymonitoring agent 306 which, in some embodiments, may include anysuitable monitoring or journaling agent as further described herein.

As shown, system 300 may also include an administrative file systemprogram 316, such as a file system program known in the art which mayinclude operating system programs, a FAT, an NTFS, or the like that maybe used to manage data movement to and from mass storage device 318.Thus, in operation, data may be written from memory 302 to mass storagedevice 318 via file system program 316. Such an operation may occur, forexample, to access data used to service an application running on acomputing device. During this operation, monitor agent 306 may capturethis interaction and generate a record indicating that an interactionhas occurred and store the record in index 310. The data may be storedin mass storage 318 under the supervision of file system manager 316.

As shown in FIG. 3A, monitor agent 306 may analyze data interactionssuch as interactions between memory 302 and mass storage 318 via thefile system manager 316, and record any such interactions in monitorindex 310. Thus, as described above, monitor index 310 may represent alist of data interactions wherein each entry represents a change thathas occurred to client data along with certain information regarding theinteraction. In embodiments where Microsoft Change Journal or othersimilar software is used, such entries may include a unique identifiersuch as an update sequence number (USN), certain change journal reasoncodes identifying information associated with a reason(s) for the changemade, along with data or metadata describing the data and certain dataproperties, data copy types, combinations of the same or the like.

Thus, in operation, as data moves from memory 302 to mass storage 318(or vice versa), monitor agent 306 may create and write an entry toindex 310 which may in turn, be analyzed and classified byclassification agent 312 b for entry in metabase 314. In someembodiments, classification agent 312 a may be coupled with mass storagedevice (either directly or through file system manager 316) and writemetadata entries to both metabase 314 and mass storage device 318. Insome embodiments, the metabase information may be stored on mass storagedevice 318. Moreover, in certain embodiments, classification agent 312 bmay periodically copy or backup metabase 314 to a device under thedirection of a storage manager and/or pursuant to a storage policy suchthat the information in metabase 314 may be quickly restored if lost,deleted or is otherwise unavailable.

In some embodiments, optional classification agent 312 a may operate inconjunction with monitor agent 306 such that data moving to mass storagedevice 318 is classified as further described herein and written todevice 318. With this arrangement, the data, along with the processedmetadata describing that data, is written to mass storage device 318.This may occur, for example in embodiments in which monitor agent 306and classification agent 312 a are combined into update agent 304.Writing metadata in this way allows it to be recovered or accessed frommass storage device 318 if necessary, for example, when metabase 314 ismissing certain information, busy, or otherwise inaccessible.

In certain embodiments, content agent 315 may be generally used toobtain or filter data relating to content of the data moving from memory302 to mass storage 318. For example, content agent 315 may read datapayload information and generate metadata based on the operation forstorage in metabase 314 and may include a pointer to the data item inmass storage 318. The pointer information may optionally be stored in anindex. This metadata may also be stored with the data item in massstorage 318 or as an entry functioning in place of or in addition tometabase 314. Storing metadata relating to data content in metabase 314provides the capability to perform content searches for data in themetabase 314, instead of searching entries in mass storage 318. Thisallows the system to quickly locate information satisfying a contentquery that may be retrieved from mass storage 318, if necessary.

Moreover, such content metadata may be generated and used in locatingdata based on content features throughout a hierarchy within a storageoperation system (e.g., content metadata may be generated and stored ateach or certain levels of storage within the system (primary, secondary,tertiary, and the like) to facilitate location and retrieval of databased on content). As will be understood by one of skill in the art, thefunctionality provided by the content agent 315, classification agent312 a and 312 b and monitor agent 306 may be provided by one or moremodules or components such that the modules may be integrated into asingle module providing the described functions, or may be implementedin one more separate modules each providing some of the functions.

FIG. 3B is a flow chart 350 illustrating some of the blocks that may beinvolved in the journaling process described above, according to certainembodiments of the disclosure. At block 355, the monitor program may beinitialized, which may include instantiating a data structure or indexfor recording interaction entries, and the assignment of a uniquejournal ID number which allows the system to differentiate betweenvarious journaling data structures that may be operating within thesystem. As mentioned above, the monitor program may include a filterdriver or other application that monitors data operations (block 360).During the monitoring process, the monitor agent may observe datainteractions between memory and mass storage to determine that certaindata interactions have occurred. Information relating to theinteractions may be captured and used to populate the metabase. In someinstances, interaction types or certain aspects of interactions arecaptured. Such types or aspects may be defined in an interactiondefinition, which may be a change journal reason codes as used byMicrosoft's Change Journal program, or be defined by a user or networkadministrator to capture some or all data interactions in order to meetuser needs. For example, certain change definitions may record everydata interaction that occurs regardless of whether any data actuallychanges or not. Such information may be useful, for example, todetermine users or processes that have “touched,” scanned or otherwiseaccessed data without actually changing it.

Thus, it is possible to employ interaction definitions that may capturea relatively broad or narrow set of operations, allowing a user tocustomize the monitor program to meet certain goals. Such interactiondefinitions may define or describe data movement, changes, manipulationsor other operations or interactions that may be of interest to a systemuser or administrator (e.g., any operation that “touches” data may berecording along with the action or operation that caused the interaction(e.g. read, write, copy, parse, or the like.) Moreover, changedefinitions may evolve over time or may be dynamic based on the entriessent to the index. For example, if expected results are not obtained,change definitions may be modified or additional definitions used untilappropriate or desired results are obtained. This may be accomplished,for example by globally linking certain libraries of interactiondefinitions and selectively enabling libraries on a rolling basis untilacceptable results are achieved. This process may be performed after theinitial activation of the monitor agent and periodically thereafter,depending on changing needs or objectives.

Moreover, in some embodiments, the system may support the use of “usertags” that allow certain types of information to be tagged so they maybe identified and tracked throughout the system. For example, a user maydesignate a particular type of data or information such as projectinformation, or information shared between or accessed by particulargroup of users to be tracked across the system or through various levelsof storage. This may be accomplished through a user interface thatallows a user to define certain information to be tagged, for example,by using any available attribute within the system such as thosespecified above with respect to the classification agent or filter usedin the system. In some embodiments, the user may define one or more tagsusing these or other attributes which may be further refined bycombining them through the use of logical or Boolean operators to adefine a certain tag expression.

For example, a user may define a certain tag by specifying one or morecriteria to be satisfied such as certain system users, a certain datapermission level, a certain project, combinations of the same or thelike. These criteria may be defined using a logical operators such as anAND or OR operators to conditionally combine various attributes tocreate a condition that defines a tag. In certain embodiments,information satisfying those criteria may be tagged and tracked withinthe system. For example, as data passes through the monitor agent 306(or other module within update agent 304), the data satisfying thesecriteria may be identified and tagged with a header or a flag or otheridentifying information as is known in the art. This information may becopied or otherwise noted by metabase 314 and mass storage 318 so thatthe information may be quickly identified. For example, the metabase 314may contain entries keeping track of entries satisfying the tag criteriaalong with information relating to the types of operations performed onthe information as well as certain metadata relating to the data contentand its location in mass storage 318. This allows the system to searchthe metabase 314 at a particular level of storage for the information,and quickly locate it within mass storage device for potentialretrieval.

Next, a block 365, the monitor agent 306 may continue to monitor datainteractions based on change definitions until an interaction satisfyinga definition occurs. Thus, a system according to one embodiment of thepresent disclosure may continue to monitor data interactions at blocks360 and 365 until a defined interaction, such as an interactionsatisfying or corresponding to a selection criteria, such as aninteraction definition or the like, occurs. If a defined interactiondoes occur, the monitor agent 306 may create a record, which may bestored in a monitor program index, and in some embodiments, assign aninteraction code that describes the interaction observed on the dataobject. Next, a block 370, the monitor program may identify a dataobject identifier associated with the data and that is associated withthe data interaction, such as a file reference number (FRN) related tothe data object. The FRN may include certain information such as thelocation or path of the associated data object. Any additionalinformation (e.g., data properties, copy properties, storage policyinformation, combinations of the same or the like) associated with theFRN may also be obtained in order to enrich or enhance the record. Insome embodiments, this may further involve obtaining information fromother system files including master file tables (MFTs) to furtherenhance the metabase entries. Additional processing or formatting of themetabase entries may also occur in accordance with certain definedclassification paradigms in order to populate the metabase with optimalor preferred information.

Next, at block 375, the record may be assigned a record identifier suchas, for example, a unique update sequence number (USN) that may be usedto uniquely identify the entry within the index, and, in someembodiments, act as an index memory location. Thus, a particular recordmay be quickly located with a data structure based on the USN. Next atblock 380, the information described above may be concatenated orotherwise combined with other data or metadata data obtained by themonitor agent and arranged in an appropriate or expected format tocreate the record that may be written to the monitor index.

In other embodiments, the information described above may be written tothe index and arranged at the index into an expected format or may bewritten to the record “as received” and include metadata or otherinformation, such as a header describing that information such thatadherence to a strict data format is not required. For example, somerecords may contain more or less information than other records, asappropriate. After the record has been constructed and deemed complete,the record may be “closed” by the system at block 385 and the system maythen assign another USN to the next detected change. However, if therecord is determined to be incomplete, the monitor agent or update agentmay request any missing information to complete the entry. If suchinformation is not received, the monitor agent may set a flag within therecord to indicate it contains incomplete information and the record maythen be closed.

FIG. 4 is a flow chart 400 illustrating some of the blocks that may beinvolved in a data classification process, according to certainembodiments of the present disclosure. At block 410, the classificationagent may be initialized, which may include activating, reserving and/orclearing certain buffers and/or linking to libraries associated withdeployment of the classification agent. Prior to scanning theinteraction records generated by the monitor agent as described above,the classification agent may classify existing stored data by, forexample, traversing the file and directory structure of an object systemto initially populate the metabase as described herein.

Next, at block 420, during normal operation, the classification agentmay scan the entries in the interaction journal to determine whether newentries exist since any previous classification processing wasperformed, for example, by determining whether the most recent entrycurrently in the journal is more or less recent than the last journalentry analyzed. This may be accomplished in several ways. One method mayinclude scanning a time or date information associated with the lastjournal entry examined and comparing it to the most recent time or dateinformation than the entry currently present in the journal. If it isdetermined that the most recent journal entry occurred after a previousclassification process, this process may be performed iteratively by“walking backwards” through the journal entries until the last entrypreviously analyzed by the classification agent is found. Entries withtime information after that point may be considered new or unprocessedby the classification agent (block 440). If the last entry analyzed hasthe same time stamp as the most recent entry in the journal, no newentries exist and the system may return to block 420 to continuemonitoring.

Another method of identifying new journal entries may include comparingrecord identifiers such as USN numbers assigned to each journal entry(block 430). Journal entries with a larger USN number than the lastentry previously analyzed may be considered new or unprocessed. If thelast entry analyzed has the same USN number as the current entry, no newentries exist and the system may return to block 420 to continuemonitoring. This comparison may be performed until new entries arelocated (block 440) or until it is determined that no new entries exist.

In other embodiments, rather than scanning the journal data structurefor new entries, any entries created by the journal agent may beautomatically sent to the classification agent and the identificationprocess may be largely unnecessary (except for the case where suchscanning is necessary or desirable, for example, to repopulate thejournal or verify certain information).

Next, at block 450, assuming new journal entries are identified, thesystem may determine if a metabase record already exists for the dataobject associated with those entries. This may be accomplished bycomparing data object identifiers, such as FRNs of metabase entries withdata object identifiers such as FRNs of journal entries. Matching theseand other unique data characteristics may be used to match or correlatemetabase and journal entries.

If no corresponding metabase record is identified, a new record may becreated at block 460. This may involve creating a new metabase entry ID,analyzing the journal entry and parsing the entry into a predeterminedformat, and copying certain portions of the parsed data to the newmetabase entry (blocks 460 and 470), as further described herein. Anyadditional metadata or file system information may also be associatedwith the new entry to enhance its content, including information from anFRN or information derived from an interaction code present in thejournal entry, file system such as, for example, MFT (block 480).

On the other hand, if a corresponding metadata entry is identified, thenew journal entry may be processed as described above and may overwritesome or all of the corresponding entry. Such an updated pre-existingentry may receive an updated time stamp to indicate a current revision.However, in some embodiments, even if a corresponding entry is located,a new entry may be created and written to the metabase and optionallyassociated with the existing record. In such a system, the older relatedrecord may be maintained, for example, archival, recreation, historicalor diagnostic purposes, and in some embodiments, may be marked orindicated as outdated or otherwise superseded. Such correspondingentries may be linked to one another via a pointer or other mechanismsuch that entries relating to the history of a particular data objectmay be quickly obtained.

Next, at block 490 the system may process any additional new journalentries detected by returning to block 450, where those additional newentries may be processed as described above. If no new entries aredetected, the system may return to block 420 to perform additional scanson the journal data structure and continue monitoring.

FIG. 5 illustrates an embodiment of the present disclosure in which asecondary processor performs some or all of functions associated withthe data classification process described herein, including certainsearch functions. As shown, system 500 may include a manager module 505which may include an index 510, a first computing device 515, (which mayinclude a first processor 520, a journal agent 530, and a dataclassification agent 535), and a second computing device 540 which mayinclude a second processor 545 and a data classification agent 535.System 500 may also include data store 550, a metabase 555 and changejournal 560.

Computing devices 515 and 540 may be any suitable computing device asdescribed herein and may include clients, servers or other networkcomputers running software, such as applications or programs thatcreate, transfer, and store electronic data. In some embodiments,metabase 555 and journal 560 may be physically located within computingdevice 515, e.g., stored on local mass storage. In other embodiments themetabase 555 and journal 560 may be external to computing device 515 (ordistributed between the two). In yet other embodiments, metabase 555 isaccessible via a network and journal 560 is a local device.

In operation, computing device 515 may operate substantially similarsystem 300 shown in FIG. 3 with second processor 545 in second computingdevice 540 performing certain functions. For example, as shown, dataclassification agent 535 and journaling agent 530 may operatesubstantially as described in connection with FIG. 3, i.e., journalingagent monitors data interactions on computing device 515 and records theinteractions in journal 535 and classification agent processes journalentries and populates metabase 555.

However, certain of the functions may be initiated or performed in wholeor in part by second processor 545. Computing operations associated withjournal agent 530 and/or classification agent 535 may run on or bedirected by second processor 545 and may also utilize support resourceslocated on or associated with computing device 540 such that theresources on computing device 515 are substantially unimpacted by theseoperations. This may serve to offload certain non-critical tasks fromthe host system (computing device 515) and have them performed by asecondary computing device (computing device 540).

For example, in some embodiments, the processing burden associated withsome or all of the following tasks normally performed by first computingdevice 515 may be performed by processor 545 and associated resources insecond computing device 540: (1) the initial scan of client files by theclassification agent 535 and population of metabase 555, (2) the ongoingmonitoring of data interactions of computing device (e.g., 515) andgeneration of interaction records for storage in journal 560, (3)processing and classification of journal information for updatingmetabase 555; and (4) searching or otherwise analyzing or accessingmetabase 555 and/or journal 560 for certain information. However, insome embodiments it may be preferred to assign the secondary computingdevice the certain tasks such as those associated with searchingmetabase 555, while other tasks such as updating the journal andmetabase may be performed by the primary computing device.

Performing such operations using a secondary or other processor may bedesirable, for example, when the primary processor (e.g., processor 520)is unavailable, over utilized, unavailable or otherwise heavily used, orwhen it is otherwise desired to remove the primary processor and otherprimary system resources from performing certain tasks such as the onesdescribed above. For example, in the case where it is desired to searchor access metabase 555 for certain information, it may be preferable toavoid using processor 520 for this task so it remains free to performother tasks associated with programs operating on computing device 515(e.g., when computing device 515 is busy performing other network orapplication-related functions).

In some embodiments, the secondary processor may be located on computingdevice 515 (e.g., processor 525) and may perform the operationsdescribed herein in connection with processor 545. Moreover, someembodiments may include a manager module 505 which may coordinateoverall operations between the various computing devices. For example,manager module 505 may monitor or otherwise be cognizant of theprocessing load on each computing device and may assign processing tasksbased on availability (e.g., load balance). For example, if processor520 is idle or operating at a low capacity, a request to search metabase555 may be handled by processor 520. However, if processor 520 is busyor scheduled to perform or is performing priority work, manager 505 mayassign the task to processor 545. Manager 505 may act as a generalarbiter for such processor assignments to ensure system 500 is makingefficient use of system resources.

FIG. 6 is flow chart 600 illustrating some of the blocks involved inperforming a query on a metabase in a multiple processor system similarto the system shown in FIG. 5, according to certain embodiments of theinvention. At block 610, a query may be received by the system forcertain information. This request may be processed and analyzed by amanager module or other system process (block 620) that determines orotherwise identifies which metabase or metabases within the systemlikely include at least some of the requested information (block 630).For example, the query itself may suggest which metabases to searchand/or the management module may consult an index that containsinformation regarding metabase content within the system as furtherdescribed herein. It will be understood that the identification processmay require searching and identifying multiple computing devices withinan enterprise or network that may contain information satisfying searchcriteria.

In other embodiments, search requests may be automatically referred to asecondary processor to minimize processing demands on the computingdevice that may have created or is otherwise associated with theidentified metabase(s). In some embodiments, it is preferable that thecomputing device that created or is otherwise associated with theidentified metabase(s) not be involved in processing search operationsas further described herein. Thus, the secondary computing device mayconsult with a manager or index associated with other computing devicesto identify metabases with responsive information.

Next at block 640, the secondary processor may search metabases toidentify appropriate data set that may potentially have informationrelated to the query. This may involve performing iterative searchesthat examine results generated by previous searches and subsequentlysearching additional, previously unidentified metabases to findresponsive information that may not have been found during the initialsearch. Thus the initial metabase search may serve as a starting pointfor searching tasks that may be expanded based on returned or collectedresults. Next, at block 650, the returned results may be optionallyanalyzed for relevance, arranged, and placed in a format suitable forsubsequent use (e.g., with another application), or suitable for viewingby a user and reported (block 650).

FIG. 7 presents a system 700 constructed in accordance with theprinciples of the present disclosure employing a centralized metabase760 that may serve multiple computing devices 715, 720, 725. Forexample, as shown, system 700 may include computing devices 715, 720,725, each of which may include a journaling agent (730, 735, 740respectively), a classification agent (745, 750, 755 respectively), andcentralized metabase 760, and in some embodiments, a manager module 705with an index 710.

In operation, system 700 may operate substantially similar to system 300shown in FIG. 3 with each computing device 715, 720, 725 storingclassification entries in centralized metabase 760 rather than eachcomputing device having its own dedicated metabase. For example, asshown, data classification agents 745, 750, 755 may operatesubstantially as described herein and communicate results to centralizedmetabase 760. That is, the data classification agents 745, 750, 755 mayanalyze and process entries within the respective journals associatedwith journaling agents 730, 735, 740, and report results to metabase760. With this arrangement, the classification agent may provide eachmetabase entry with an ID tag or other indicia that identifies whichcomputing device 715, 720, 725 the entry originated from to facilitatefuture searches and efficiently designate entry ownership, or otherassociations between entries and computing devices.

Moreover, each entry to metabase 760 may be assigned a unique identifierfor management purposes. As mentioned above, this number may representthe index location or offset of the entry within centralized metabase760. In some embodiments, entries may be communicated to metabase 760from the computing devices 715, 720, 725 on a rolling basis and may bearranged and formatted for storage by the metabase 760. For example,metabase 760 may receive multiple entries at substantially the samepoint in time from multiple computing devices 715, 720, 725 and may beresponsible for queuing and arranging such entries for storage withinthe metabase 760.

In some embodiments, the system 700 may include manager module 705 thatmay be responsible for assigning or removing associations betweencertain computing devices 715, 720, 725 and a particular centralizedmetabase 760. For example, in accordance with certain system preferencesdefined in index 710, manager 705 may direct certain computing devices715, 720, 725 to write classification entries to a particularcentralized metabase 760. Information indicating an association of themetabase 760 and the computing devices 715, 720, 725 may be stored inthe index 710. In certain embodiments, this allows system 700 toreassign resources (globally or locally) to optimize system performancewithout the need to change device pointers or code associated with eachcomputing device 715, 720, 725 that may be affected by a particularreallocation. For example, manager 705 may reassign certain computingdevices 715, 720, 725 to another metabase by changing a destinationaddress in an appropriate index.

FIG. 8 illustrates a flow chart 800 illustrating some of the blocksinvolved in using a centralized metabase with multiple computing devicessimilar to the one shown in FIG. 7, according to certain embodiments ofthe present disclosure. At block 810, a centralized metabase may beinstantiated by a manager module or in accordance with certain systemmanagement or provisioning policies. This may involve securing certainprocessing, storage, and management resources for performing the task,loading certain routines into various memory buffers and informing themanagement module that the metabase is ready for operation.

Next, at block 820, the management module may review system resources,management policies, operating trends, and other information, forexample, to identify computing devices to associate with theinstantiated centralized metabase. This may further involve identifyingpathways to the metabase from the various computing devices, locatingoperational policies governing the computing devices and, creatingcertain logical associations between the centralized metabases and theidentified computing devices. These associations, once created, may bestored in an index or database for system management purposes.

After the metabase has been instantiated and associated with computingdevices, classification agents within each associated computing devicemay scan existing files or data on the computing devices or clients(block 825) and populate the centralized metabase as further describedherein (block 830). During the scanning process, a computing deviceidentifier or other indicia may be appended or otherwise associated withthe entry prior to transmission to the metabase such that each entry inthe metabase can be tracked to its associated source computing device(block 840). Next, the centralized metabase may be populated withentries (block 850) and may communicate with the management module toestablish and monitor a list of computing devices serviced by thecentralized metabase (block 860) and return to block 830. At this point,the system continues to monitor the associated computing devices fordata interactions, which may be reported to the centralized metabase onan ongoing, periodic, or rolling basis.

In certain circumstances, the centralized metabase may need toassimilate or otherwise integrate existing entries with new entriesreported by the computing devices. For example, the centralized metabasemay become disconnected or unavailable for a period of time andsubsequently be required to integrate a large number of queued entries.In this case, the metabase or management module may examine existingmetabase entries as described herein and communicate with computingdevices to identify: (1) the amount of time the object computer and themetabases have been disconnected, (2) the number of queued entries atthe computing devices that need to be processed (for example, entriescached once the centralized metabase was inaccessible for writeoperations), (3) whether there are any duplicative entries, and (4)which entries need to be integrated and in what order of preference(assuming multiple computing devices contain queued entries).

Based on one or more of these criteria, the management module orcentralized metabase may assimilate the relevant entries into themetabase in the appropriate order until the backlog is eliminated andthe system returns to normal operation. If it is determined during thisprocess that certain information is lost to cache overflow, accidentaldeletion, corruption, or other reasons, the metabase and/or managermodule may indicate such a discontinuity with the metadata or indexassociated with the centralized storage device or management module. Inthis case, clients, computing devices or other data sources may berescanned to replace or repair the faulty entries. In other embodiments,the points of discontinuity may be noted and interpolation or other datahealing techniques may be employed to provide derived information forthe unknown points based on known information.

FIG. 9 presents a system 900 constructed in accordance with theprinciples of the present disclosure including a computing device thatinteracts with a network attached storage device (NAS). As shown, system900 may include a management module 905 and index 910, computing devices915, 920, 925, each of which may include a journaling agent (945, 950,955 respectively), a classification agent (930, 935, 940 respectively),data stores 960 and 965, and metabases 970, 975, 980, respectively.System 900 may also include NAS device 995 which may include NAS storagedevice 990 and NAS file system manager 985. Moreover, computing device925 may be configured to operate as a NAS proxy device supervising thetransfer of data to and from NAS device 995.

In operation, system 900 may operate substantially similar to system 300shown in FIG. 3A with exception of the illustrated NAS portion. Forexample, as shown, data classification agents 930, 935, 940 may operatesubstantially as described herein and communicate results to theirrespective metabases 970, 975, 980. That is, analyze and process entrieswithin the respective journals associated with journaling agents 945,950, 955, and report results to metabases 970, 975, 980 which may besupervised in whole or in part by management module 905.

Data from computing device 925 may be journaled and classified usingmethods similar to those described herein. For example, journaling agent955 may reside on computing device 925 and track each or certain datainteractions between NAS device 995 and external applications. Thelocation of the journaling agent 955 may be external to the NAS device995 due, at least in part, to its proprietary nature (i.e., a closedsystem) and the difficulty associated with attempting to run otherprograms on the NAS device 995 itself.

The NAS portion 995 of system 900 may operate somewhat differently. Forexample computing device 925 may operate as a NAS proxy for moving datafiles to and from NAS device 995 using a specialized protocol such asthe Network Data Management Protocol (NDMP) that is an open networkprotocol designed to perform data backups over heterogeneous networks.NDMP may be used to enhance performance by transferring data over anetwork by separating data and control paths, while maintainingcentralized backup administration.

Journaling agent 955 may record any interactions between NAS data andexternal applications and record those interactions in computing device925 as described herein. In some embodiments, such a journaling agentmay include specialized routines for interpreting and processing data inNAS format. Data classification agent 940 may analyze journal entriesand populate metabase 980 initially and periodically as furtherdescribed herein.

Once initially populated, it may be desired to search the metabases ofsystem 900 for certain information. This is discussed in more detailbelow in connection with the flow chart of FIG. 11. In some embodiments,this may be handled by manager 905 or other system process which mayinitially evaluate any search request and consult index 910 or otherinformation stores to determine which metabases within the system arelikely to include responsive information. The results of this evaluationmay be provided to the computing device handling the search request andmay be in the form of pointers or other indicia or identifiersidentifying a metabase such as a metabase ID. This may allow thecomputing device posing the search request to contact and search theidentified metadata directly. In other embodiments manager 905 mayprocess the request and provide substantially complete results to thecomputing device that submitted the query.

FIG. 10 is a flow chart 1000 illustrating some of the blocks that may beinvolved in using the NAS system similar to or the same as the one shownof FIG. 9, according to certain embodiments of the present disclosure.At block 1010, a copy operation may be initiated that directs data fromcomputing device to a NAS device. This may involve identifying certaindata to moved, for example, based on a data management or storagepolicy. Other factors that may also be considered may include data size,the last time the data was moved to the NAS device, the file owner,application type, combinations of the same or the like.

It will be understood that in some embodiments it may be preferred touse computing device 925 as a NAS proxy that routes data from othernetwork computing devices to NAS device 995 with the computing device925 supervising the data movement using certain specialized transferprograms to assist in the effort (block 1020). As the data is routedthough computing device 925, journaling agent 955 may monitorinteractions with NAS device 995 and create interaction entries for aninteraction journal (block 1030). This may be accomplished by consultingwith NAS file manager 985 and identifying which files in NAS 995 thathave been involved in a data interaction as further described herein(block 1040). Next, journal entries may be created or updated to reflectdata interactions currently detected as previously described herein(block 1050). The interaction journal may then be scanned to analyze thejournal records (block 1060) and perform the classification process asfurther described herein to create metabase entries (block 1070). Atthis point metabase entries may be assigned an identifier and used topopulate metabase 980 (block 1080).

As mentioned above, under certain circumstances, it may be desired tosearch a system that includes multiple metabases for certain informationsuch as system 900 shown in FIG. 9 whether or not NAS included. FIG. 11includes a flow chart 1100 illustrating some of the blocks that may beperformed in searching a multiple metabase system in accordance withcertain embodiments of the present disclosure.

Assume, for example, a user wants to locate and copy all data relatingto a certain specified criteria such as data relating to a specificmarketing project created and edited by a specific group of users over acertain period of time. First, the requestor may formulate such arequest through a user interface using techniques known in the art andsubmit the request to the system for processing. This may also beaccomplished by an automated computerized process, for example, when thesystem is performing certain management functions. Next the system mayreceive and analyze this query (block 1110). In some embodiments, thismay be performed by a computing device configured to support the userinterface. In other embodiments, the computing device may simply passthe request to the system where a management module or other systemprocess computing device may perform the analysis. The analysis mayinclude determining characteristics of data in the metabase that maysatisfy the selected criteria.

Once the search request or query has been analyzed or parsed, the systemmay identify all metabases likely to contain records related to relevantdata objects based on a query (block 1120). This may be accomplished byusing information obtained from analyzing or parsing the request as wellas consulting with a management module that may have a substantiallyglobal view of metabases within the system that includes indexinformation or a general overview of the information the metabasescontain. After a set of metabases have been identified, the managementmodule or other computing device may perform the search to identify adata set satisfying a query as further described herein and return a setof results (block 1130). At block 1140, the results may optionally benormalized. If normalization is not required, the results may bereported at block 1150. If normalization is desired, the system mayanalyze the results for both content and completeness. If, based on thereturned results, other unsearched metabases are implicated aspotentially having information that satisfies the search criteria, thosemetabases may be searched as well. This process may continue in aniterative fashion until a substantially complete set of results isobtained. Even if no additional metabases are implicated, these resultsmay then be optionally normalized by performing certain functions suchas locating and removing duplicative results, identifying networkpathways to data objects identified in the search, and formatting orarranging the results for further processing (whether another computingprocess or for a user). For example, the returned results may be used tolocate and retrieve the responsive data objects that may includeinformation located on primary or secondary storage devices within thesystem or for other purposes as further described herein.

In some embodiments, the systems and methods of the present disclosuremay be employed to identify and track some or all data interactions on auser or group basis. For example, a system administrator or user maywish to record and keep track of data interactions involving some or allsystem groups or users. This may include, for example, read and writeoperations performed on the user's or group's behalf, information andapplications used or accessed, viewed web pages, electronic gaminginteractions, chat, instant messages, and other communicationinteractions, multimedia usage, other Internet or network basedelectronic interactions as known in the art, combinations of the same orthe like. Thus, in certain embodiments, the system identifies, captures,classifies, and may otherwise tracks user and group interactions withelectronic data creating a data store or other repository of theseinteractions and metadata associated with these interactions. In someembodiments, this repository may serve as a “digital or electronic liferecord” that effectively chronicles and catalogues some or all user orgroup interactions with electronic information and data during a giventime period as further described herein.

For example, FIG. 11A illustrates a system 1160 constructed inaccordance with the principles of the present disclosure thatidentifies, captures, classifies, and otherwise tracks user and groupinteractions with electronic data. As shown, the system 1160 maygenerally include computing device 1162, one or more classificationagents 1164, one or more journaling agents 1165, a metabase 1166, achange record 1167, and a database 1168.

In operation computing device 1162 may be coupled to or interact withvarious other applications, networks, and electronic information suchas, for example multimedia applications 1170, instant messaging/chatapplications 1172, network applications 1174 such as an enterprise WANor LAN, Internet 1176, and gaming applications 1178. It will beunderstood, however, that these are only exemplary and that any othernetwork, application, or type of electronic information suitable for thepurposes described herein may be added if desired.

Journaling agents 1165 and classification agents 1164 may operate inconjunction with one another to detect and record data interactions asfurther described herein. For example, each type electronic datainteraction (e.g., instant messaging, web surfing, Internet searchactivities, electronic gaming, multimedia usage, combinations of thesame or the like) may be identified, captured, classified, and otherwisetracked by a different journaling agent 1165 and classification agent1164, for example an interaction-specific journaling agent 1165 orclassification agent 1164 dedicated to processing a single type ofinteraction with electronic data. Thus, the system may have a firstjournaling agent 1165 and a first classification agent 1164 monitoringnetwork traffic on a given network interface (not shown) directed tointeractions associated with Internet usage, and a second journalingagent 1165 and a second classification agent 1164 monitoring a differentsystem resource directed to interactions associated with electronicgaming (e.g., recording and classifying gaming interactions such asrecording games played, opponents played, win/loss records, combinationsof the same or the like) or directed to interactions associated with useof an Internet browser to “surf” web (e.g., tracking pages visited,content, use patterns, combinations of the same or the like) In someembodiments, journaling agent 1165 and classification agent 1164 mayfunction as a single module capable of performing some or all functionsassociated with journaling agent 1165 and a classification agent 1164.

Thus, as a user or group interacts with various types of electronicinformation, some or all of those interactions may be captured andrecorded in database 1168. Change record 1167 and metabase 1166 mayrecord certain aspects of the interactions as further described hereinand may represent an interaction by interaction log of the user'scomputing activities.

For example, in operation, a user of computing device 1162 may interactwith certain applications such as multimedia application 1170 andinstant messaging application 1172. This may include sending, receiving,viewing and responding to various audio/video files in any suitableformat and may include instant, text or email messages. Journaling agent1165 may detect the interactions between these applications andcomputing device 1162 and classification agent 1164 may classify andrecord information (e.g., metadata) associated with these interactionsin metabase 1166 as further described herein.

Moreover, in some embodiments, some or all the content being exchangedor otherwise associated with these interactions may be captured andstored in database 1168 or other storage locations in the system. Thismay include capturing screen shots or summaries of information exchangesduring data interactions. For example, the system may download allcontent associated with web pages viewed thus being able to recreate theoriginal page content and interaction without access to the original orsource version of the page on the Internet or other network. This may beadvantageous, for example, if a user wishes to interact with contentassociated with a previous interaction when that content is no longeravailable, as is common with web pages and other network resources overtime. As another example, the system may also capture or otherwise storedata associated with other interactions, for example chat transcripts,video game replays, search queries, search results, and associatedsearch content, songs accessed, movies accessed, stored songs andmovies, combinations of the same or the like, in addition to metadata.

Moreover, in some embodiments, specialized classifications agents may beemployed for some or all of the applications that a user oradministrator desires to track and record. For example, the multimediaand instant messaging applications described above may each have adedicated classification agent that analyzes journal records to createentries for metabase 1166. Further still, each classification agent mayhave its own associated metabase and or repository for source data (notshown), so application histories and content may be quickly indexed andsearched. In other embodiments, however, a “universal” classificationagent may be used that recognizes the application type (e.g., based onthe journaling agent entries) and process interactions accordingly(which may include routing metadata to one or more specializedmetabases).

As shown in FIG. 11A, computing device 1162 may also interact withvarious network applications 1174 such as LAN or WAN applications. Thesemay include interaction with certain distributed programs such asMicrosoft Word or Outlook. Users may also interact with Internet 1176and download various web pages and other information. In accordance withan aspect of the present disclosure, interactions with thesenetworks/applications may also be journaled as described above withcertain information regarding these interactions stored in metabase1166. Portions of exchanged content may also be stored in database 1166.For example, Word documents, emails, web pages, web addresses and HTMLcontent may be captured and stored on database 1168 such that itsubstantially represents a record of all user interactions withcomputing device 1162, or other system devices. For example, userinteractions may be recorded with respect to any identified user basedon identifiers and tracked at any network computing device.

Thus, if desired a user may retrieve captured data and review or replaycertain data exchanges or save such records for future reference. Forexample, a user may store all instant messaging interactions for replayor transmission to another. In some instances, it may be desirable tonot record certain interactions, such as personal or privateinformation. In some embodiments, this may be accomplished by“disabling” the appropriate classification agent for a certain period oftime or the like.

Likewise, interactions with gaming applications (network or stand alone)may also be recorded stored with appropriate information stored indatabase 1168 and metabase 1166. Thus, a user may have the ability toretrieve, replay and transmit certain saved gaming sequences to thirdparties.

In some embodiments, database 1168 may become large and thus someinformation stored thereon may be moved to single instance storage fromdatabase 1168 with pointer placed in the logical address of theinstanced information (not shown). This may be performed as a memorysaving measure as at least some of the entries in database 1168 arelikely to be duplicative.

Some of the blocks associated with the method generally described aboveare illustrated in the flow chart 1200 of FIG. 12 and may include thefollowing. At the outset, a group or user of interest may be identifiedbased on certain user related information or other networkcharacteristics (block 1210). Such characteristics may include ActiveDirectory privileges, network login, machine ID, or certain biometricsassociated with a user or group member. These characteristics may becombined together or associated with one another to create a user orgroup profile. Such profiles may be stored in a database or index withina management module of the system and may be used as classificationdefinitions within the system. When it is desired to identify orclassify data items associated with a particular interaction, the systemmay compare certain attributes of the data involved in a detectedinteraction and associate that interaction with a particular group oruser based on profile information (block 1220).

Such associations may be stored in a metabase created to keep track ofuser or group interactions. Thus, in one embodiment, the metabaseessentially represents a list of all data interaction for a particulargroup or user. If desired, a list or copy of all the data items touched(e.g., interacted with) by a group or user may be quickly obtained.

In operation, the system may, through the use of a journaling agent orthe like, monitor data interactions for a particular computing device asdescribed herein. The interactions may be analyzed by a classificationagent as described herein and associated with one or more profiles(block 1230). The association may be recorded in an identifiedmetabase(s) that keeps track of a user's or group's interactions (block1240) which may include references to the data object(s) identified, theattributes compared, and the basis for the association, combinations ofthe same or the like. As discussed herein, the journaling agent maycontinue to monitor data interactions throughout operation, so that eachmetabase is updated and continues to accurately represent the datatouched by a particular group or user. The identified metabases areassociated with an identified group or user (block 1250), such as bystoring an indication of the association in an index.

FIG. 13 presents a system 1300 constructed in accordance with theprinciples of the present disclosure for communicating metadata and/ordata objects between two or more computing devices. As shown, system1300 may generally include first and second computing devices 1310 and1320, respectively, associated data stores 1330 and 1340, and metabases1350 and 1360. Computing devices in system 1300 may store data objectsand metadata in their respective metabases and data stores as furtherdescribed herein. In certain situations, however, it may be desired totransfer certain metadata between metabases 1350 and 1360 and certaindata objects between data stores 1330 and 1340. This may be desirablefor example, to move certain data from one computing device to another,to recreate a certain application at another location, or to copy orbackup certain data objects and associated metadata.

FIG. 14 presents a flow chart 1400 illustrating some of the blocksassociated with moving data between the computing devices describedabove, according to certain embodiments of the present disclosure.First, at block 1410, data objects and/or associated metadata may beidentified for movement from one computing device to another. This maybe accomplished by forming a query for certain data, such as a searchfor data that may be to be moved or copied pursuant to a data managementor storage policy, or in response to a request to move data relating tocertain processes or applications from one computing device to another,for any other suitable purpose such as disaster recovery, resourcereallocation or reorganization, load balancing, combinations of the sameor the like.

At block 1420, the query may be analyzed and a first data storeassociated with a first computer may be searched for data objectssatisfying the search criteria. Data objects identified during thisprocess may then be transferred to a second data store associated with asecond computing device (block 1430). Metadata associated with thetransferred data objects may also be identified in a first metabaseassociated with the first computing device and transferred to anappropriate second metabase associated with the second computing device(block 1440). Such a transfer may involve copying data objects andmetadata from one data store and metabase to another, or in someembodiments, may involve migrating the data from its original locationto a second location and leaving a pointer or other reference to thesecond location so the moved information may be quickly located frominformation present at the original location.

FIG. 15 illustrates one arrangement of resources in a computing networkin accordance with the principles of the present disclosure. As shown,storage operation cell 1550 may generally include a storage manager1501, a data agent 1595, a media agent 1505, a storage device 1515, and,in some embodiments, may include certain other components such as aclient 1585, a data or information store 1590, database or index 1511,jobs agent 1520, an interface module 1525, and a management agent 1530.Such system and elements thereof are exemplary of a modular storagesystem such as the CommVault QiNetix® system, and also the CommVaultGALAXY backup system, available from CommVault Systems, Inc. ofOceanport, N.J., and further described in U.S. patent application Ser.No. 09/610,738, which is incorporated herein by reference in itsentirety.

A storage operation cell, such as cell 1550, may generally includecombinations of hardware and software components associated withperforming storage operations on electronic data. According to someembodiments of the present disclosure, storage operations cell 1550 maybe related to backup cells and provide some or all of the functionalityof backup cells as described in U.S. patent application Ser.No.09/354,058, which is hereby incorporated by reference in itsentirety. However, in certain embodiments, storage operation cells mayalso perform additional types of storage operations and other types ofstorage management functions that are not generally offered by backupcells.

In accordance with certain embodiments of the present disclosure,additional storage operations performed by storage operation cells mayinclude creating, storing, retrieving, and migrating primary storagedata (e.g., 1590) and secondary storage data (which may include, forexample, snapshot copies, backup copies, HSM copies, archive copies, andother types of copies of electronic data) stored on storage devices1515. In some embodiments, storage operation cells may also provide oneor more integrated management consoles for users or system processes tointerface with in order to perform certain storage operations onelectronic data as further described herein. Such integrated managementconsoles may be displayed at a central control facility or severalsimilar consoles distributed throughout multiple network locations toprovide global or geographically specific network data storageinformation.

In some embodiments, storage operations may be performed according tovarious storage preferences, for example as expressed by a userpreference or storage policy. A storage policy is generally a datastructure or other information source that includes a set of preferencesand other storage criteria associated with performing a storageoperation. The preferences and storage criteria may include, but are notlimited to, a storage location, relationships between system components,network pathway to utilize, retention policies, data characteristics,compression or encryption requirements, preferred system components toutilize in a storage operation, other criteria relating to a storageoperation, combinations of the same and the like. Thus, in certainembodiments, a storage policy may indicate that certain data is to bestored in a specific storage device, retained for a specified period oftime before being aged to another tier of secondary storage, copied tosecondary storage using a specified number of streams. A storage policymay be stored in the storage manager database 1511, to archive media asmetadata for use in restore operations or other storage operations, orto other locations or components of the system.

In certain embodiments, a schedule policy may specify when to performstorage operations and how often and may also specify performing certainstorage operations on sub-clients of data and how to treat thosesub-clients. A sub-client may represent static or dynamic associationsof portions of data of a volume and are typically mutually exclusive.Thus, in certain embodiments, a portion of data may be given a label andthe association is stored as a static entity in an index, database orother storage location used by the system. Sub-clients may also be usedas an effective administrative scheme of organizing data according todata type, department within the enterprise, storage preferences,combinations of the same or the like.

For example, an administrator may find it preferable to separate e- maildata from financial data using two different sub-clients havingdifferent storage preferences, retention criteria, or the like. Storageoperation cells may contain not only physical devices, but also mayrepresent logical concepts, organizations, and hierarchies. For example,a first storage operation cell 1550 may be configured to perform a firsttype of storage operations such as HSM operations, which may includebackup or other types of data migration, and may include a variety ofphysical components including the storage manager 1501 (or managementagent 1530), the media agent 1505, the client component 1585, and othercomponents as described herein. A second storage operation cell maycontain the same or similar physical components, however, it may beconfigured to perform a second type of storage operations such as SRMoperations, and may include as monitoring a primary data copy orperforming other known SRM operations.

Thus, as can be seen from the above, although the first and secondstorage operation cells are logically distinct entities configured toperform different management functions (i.e., HSM and SRM respectively),each cell may contain the same or similar physical devices in bothstorage operation cells. Alternatively, in other embodiments, differentstorage operation cells may contain some of the same physical devicesand not others. For example, a storage operation cell configured toperform SRM tasks may contain the media agent 1505, client 1585, orother network device connected to a primary storage volume, while astorage operation cell configured to perform HSM tasks may insteadinclude a media agent 1505, client 1585, or other network deviceconnected to a secondary storage volume and not contain the elements orcomponents associated with and including the primary storage volume.These two cells, however, may each include a different storage managerthat coordinates storage operations via the same media agents 1505 andstorage devices 1515. This “overlapping” configuration allows storageresources to be accessed by more than one storage manager 1501 such thatmultiple paths exist to each storage device 1515 facilitating failover,load balancing and promoting robust data access via alternative routes.

Alternatively, in some embodiments, the same storage manager 1501 maycontrol two or more cells 1550 (whether or not each storage cell 1550has its own dedicated storage manager). Moreover, in certainembodiments, the extent or type of overlap may be user-defined (e.g.,through a control console) or may be automatically configured tooptimize data storage and/or retrieval.

Data agent 1595 may be a software module or part of a software modulethat is generally responsible for copying, archiving, migrating, andrecovering data from client computer 1585 stored in an information store1590 or other memory location. Each client computer 1585 may have atleast one data agent 1595 and the system can support multiple clientcomputers 1585. In some embodiments, data agents 1595 may be distributedbetween client 1585 and storage manager 1501 (and any other intermediatecomponents) or may be deployed from a remote location or its functionsapproximated by a remote process that performs some or all of thefunctions of data agent 1595.

Embodiments of the present disclosure may employ multiple data agents1595 each of which may backup, migrate, and recover data associated witha different application. For example, different individual data agents1595 may be designed to handle Microsoft Exchange data, Lotus Notesdata, Microsoft Windows 2000 file system data, Microsoft ActiveDirectory Objects data, and other types of data. Other embodiments mayemploy one or more generic data agents 1595 that can handle and processmultiple data types rather than using the specialized data agentsdescribed above.

If a client computer 1585 has two or more types of data, one data agent1595 may be required for each data type to copy, archive, migrate, andrestore the client computer 1585 data. For example, to backup, migrate,and restore all of the data on a Microsoft Exchange 2000 server, theclient computer 1585 may use one Microsoft Exchange 2000 Mailbox dataagent 1595 to backup the Exchange 2000 mailboxes, one Microsoft Exchange2000 Database data agent 1595 to backup the Exchange 2000 databases, oneMicrosoft Exchange 2000 Public Folder data agent 1595 to backup theExchange 2000 Public Folders, and one Microsoft Windows 2000 File Systemdata agent 1595 to backup the client computers 1585 file system. In suchembodiments, these data agents 1595 may be treated as four separate dataagents 1595 by the system even though they reside on the same clientcomputer 1585.

Alternatively, other embodiments may use one or more generic data agents1595, each of which may be capable of handling two or more data types.For example, one generic data agent 1595 may be used to back up, migrateand restore Microsoft Exchange 2000 Mailbox data and Microsoft Exchange2000 Database data while another generic data agent may handle MicrosoftExchange 2000 Public Folder data and Microsoft Windows 2000 File Systemdata, or the like.

Data agents 1595 may be responsible for arranging or packing data to becopied or migrated into a certain format such as an archive file whichis discussed in more detail in connection with FIG. 16 herein.Nonetheless, it will be understood this represents only one example andany suitable packing or containerization technique or transfermethodology may be used if desired. Such an archive file may include alist of files or data objects copied in metadata, the file and dataobjects themselves. Moreover, any data moved by the data agents may betracked within the system by updating indexes associated appropriatestorage managers or media agents.

Generally speaking, storage manager 1501 may be a software module orother application that coordinates and controls storage operationsperformed by storage operation cell 1550. Storage manager 1501 maycommunicate with some or all elements of storage operation cell 1550including client computers 1585, data agents 1595, media agents 1505,and storage devices 1515, to initiate and manage system backups,migrations, and data recovery.

Storage manager 1501 may include a jobs agent 1520 that monitors thestatus of some or all storage operations previously performed, currentlybeing performed, or scheduled to be performed by storage operation cell1550. Jobs agent 1520 may be communicatively coupled with an interfaceagent 1525 (typically a software module or application). Interface agent1525 may include information processing and display software, such as agraphical user interface (“GUI”), an application program interface(“API”), or other interactive interface through which users and systemprocesses can retrieve information about the status of storageoperations. Through interface 1525, users may optionally issueinstructions to various storage operation cells 1550 regardingperformance of the storage operations as described and contemplated bythe present disclosure. For example, a user may modify a scheduleconcerning the number of pending snapshot copies or other types ofcopies scheduled as needed to suit particular needs or requirements. Asanother example, a user may employ the GUI to view the status of pendingstorage operations in some or all of the storage operation cells in agiven network or to monitor the status of certain components in aparticular storage operation cell (e.g., the amount of storage capacityleft in a particular storage device).

Storage manager 1501 may also include a management agent 1530 that istypically implemented as a software module or application program. Ingeneral, management agent 1530 provides an interface that allows variousmanagement components 1501 in other storage operation cells 1550 tocommunicate with one another. For example, assume a certain networkconfiguration includes multiple cells 1550 adjacent to one another orotherwise logically related in a WAN or LAN configuration (not shown).With this arrangement, each cell 1550 may be connected to the otherthrough each respective interface agent 1525. This allows each cell 1550to send and receive certain pertinent information from other cells 1550including status information, routing information, information regardingcapacity and utilization, or the like. These communication paths mayalso be used to convey information and instructions regarding storageoperations.

For example, a management agent in a first storage operation cell maycommunicate with a management agent in a second storage operation cellregarding the status of storage operations in the second storageoperation cell. Another illustrative example includes the case where amanagement agent in first storage operation cell communicates with amanagement agent 150 in a second storage operation cell to control thestorage manager 1501 (and other components) of the second storageoperation cell via the management agent 1530 contained in the storagemanager 1501.

Another illustrative example is the case where management agent 1530 inthe first storage operation cell 1550 communicates directly with andcontrols the components in the second storage management cell 1550 andbypasses the storage manager 1501 in the second storage management cell.If desired, storage operation cells 1550 can also be organizedhierarchically such that hierarchically superior cells control or passinformation to hierarchically subordinate cells or vice versa.

Storage manager 1501 may also maintain an index, a database, or otherdata structure 1511. The data stored in database 1511 may be used toindicate logical associations between components of the system, userpreferences, management tasks, media containerization and data storageinformation or other useful data. For example, the storage manager 1501may use data from database 1511 to track logical associations betweenmedia agent 1505 and storage devices 1515 (or movement of data ascontainerized from primary to secondary storage).

Generally speaking, a media agent, which may also be referred to as asecondary storage computing device 1505, may be implemented as softwaremodule that conveys data, as directed by storage manager 1501, between aclient computer 1585 and one or more storage devices 1515 such as a tapelibrary, a magnetic media storage device, an optical media storagedevice, or any other suitable storage device. In one embodiment,secondary computing device 1505 may be communicatively coupled with andcontrol a storage device 1515. A secondary computing device 1505 may beconsidered to be associated with a particular storage device 1515 ifthat secondary computing device 1505 is capable of routing and storingdata to particular storage device 1515.

In operation, a secondary computing device 1505 associated with aparticular storage device 1515 may instruct the storage device to use arobotic arm or other retrieval means to load or eject a certain storagemedia, and to subsequently archive, migrate, or restore data to or fromthat media. Secondary computing device 1505 may communicate with astorage device 1515 via a suitable communications path such as a SCSI orfiber channel communications link. In some embodiments, the storagedevice 1515 may be communicatively coupled to a data agent 105 via astorage area network (“SAN”).

Each secondary storage computing device 1505 may maintain a index, adatabase, or other data structure 1506 which may store index datagenerated during backup, migration, and restore and other storageoperations as described herein. For example, performing storageoperations on Microsoft Exchange data may generate index data. Suchindex data provides a secondary computing device 1505 or other externaldevice with a fast and efficient mechanism for locating data stored orbacked up. Thus, in some embodiments, a secondary storage computingdevice index 1506, or a storage manager database 1511, may store dataassociating a client 1585 with a particular secondary computing device1505 or storage device 1515, for example, as specified in a storagepolicy, while a database or other data structure in secondary computingdevice 1505 may indicate where specifically the client 1585 data isstored in storage device 1515, what specific files were stored, andother information associated with storage of client 1585 data. In someembodiments, such index data may be stored along with the data backed upin a storage device 1515, with an additional copy of the index datawritten to index cache in a secondary storage device. Thus the data isreadily available for use in storage operations and other activitieswithout having to be first retrieved from the storage device 1515.

Generally speaking, information stored in cache is typically recentinformation that reflects certain particulars about operations that haverecently occurred. After a certain period of time, this information issent to secondary storage and tracked. This information may need to beretrieved and uploaded back into a cache or other memory in a secondarycomputing device before data can be retrieved from storage device 1515.In some embodiments, the cached information may include informationregarding format or containerization of archive or other files stored onstorage device 1515.

In some embodiments, certain components may reside and execute on thesame computer. For example, in some embodiments, a client computer 1585,such as a data agent 1595 or a storage manager 1501, coordinates anddirects local archiving, migration, and retrieval application functionsas further described in U.S. patent application Ser. No.09/610,738. Thisclient computer 1585 can function independently or together with othersimilar client computers 1585.

Moreover, as shown in FIG. 15, clients 1585 and secondary computingdevices 1505 may each have associated indices and metabases (1525 and1560, respectively). However, in some embodiments each “tier” ofstorage, such as primary storage, secondary storage, tertiary storage,or the like, may have multiple or a centralized metabase, as describedherein. For example, in FIG. 15, rather than a separate metabase 1525associated with each client 1585, the metabases on this storage tier maybe centralized as discussed further herein. Similarly, second and othertiers of storage may have either centralized or distributed metabases.Moreover, mixed architectures systems may be used if desired, that mayinclude a first tier centralized metabase system coupled to with asecond tier storage system having distributed metabases and vice versa.

Moreover, in operation, a storage manager 1501 or other managementmodule may keep track of certain information that allows the storagemanager to select, designated or otherwise identify metabases to besearched in response to certain queries as further described herein.Movement of data between primary and secondary storage may alsoinvolvement movement of associated metadata and other trackinginformation as further described herein.

FIG. 15A shows an example configuration of a computing network 1570,where a plurality of storage operation cells 1550 can be managed by amaster storage manager 1572. In one embodiment, each of the storageoperation cells 1550 can be configured and operate in a manner similarto that described above in reference to FIG. 15. In one embodiment, themaster storage manager 1572 can be configured to provide management ofinter-cell operations involving data management. In one embodiment, theinter-cell operations can involve at least some functionalities that aresimilar to those associated with the storage managers 1501 and one ormore metabases described above in reference to FIG. 15.

FIG. 15B shows another example configuration of a computing network1574, where a master storage manager 1572 can manage one or more storageoperation cells 1501 and 1576. The example storage operation cells 1576aand 1576b are depicted as being variations of the storage operation cell1501 described above in reference to FIG. 15. For example, the storageoperation cell 1576a is depicted as not having the second client 1585and its associated metabase 1525 and storage device 1590 (see FIG. 15).In another example, the storage operation cell 1576b is depicted as nothaving the second secondary storage computing device 1505 and itsassociated metabase 1560 and storage device 1515 (see FIG. 15). Otherconfigurations are possible.

FIG. 15B further shows that the master storage manager 1572 can also beconfigured to provide an inter-system management involving a system 1300having a plurality of computing devices 1310. The system 1300 can besimilar to that described above in reference to FIG. 13. In oneembodiment, the inter-system operations can involve at least somefunctionalities that are similar to those associated with the storagemanagers 1501 and one or more metabases described above in reference toFIG. 15, and/or various functionalities described above in reference toFIG. 13.

FIG. 16 is a diagram illustrating one arrangement of data that may beused in constructing an archive file according to one embodiment of thepresent disclosure. As shown, archive file 1600 may include a headersection 1610, an index section 1620 and a payload section 1630. Such anarchive file may be constructed by a data agent at a client computingdevice when migrating data, for example, from primary to secondarystorage, primary storage to other primary storage, or the like. Thepayload section 1610 may include the data objects that are to be movedfrom a first location to a second location within the system (e.g.,primary to secondary storage). These data objects may be identified by adata agent and designated to be moved pursuant to a storage preferencesuch as a storage policy, a user preference, or the like. In certainembodiments, header 1610 may include routing and path information thatidentifies the origin and destination of the payload data and mayinclude other information such as a list of files copied, checksums,combinations of the same or the like. In certain embodiments, indexsection 1620 may include certain other information regarding the payloaddata objects such as size, file type, and any offset or other logicalindexing information that may been tracked by a storage managementcomponent or other component previously managing the data objects in thepayload.

In some embodiments, storage managers may index information regardingarchive files and related payload by time and storage on certain mediaso the archive files can be quickly located and/or retrieved. Forexample, it may be desired to identify certain data based on a query.The query may be analyzed and a certain time frame of interest may beidentified. The system may use this information as a basis for a querysearch of certain index information (e.g., only search for recordsconcerning operations that occurred during a specific time). In certainembodiments, this streamlines the search and retrieval process bynarrowing the universe of data needs to be searched to locate responsiveinformation.

FIG. 17 presents a flow chart 1700 that illustrates some of the blocksthat may be performed in moving data from primary storage to otherstorage devices within the system. First, at block 1702, a query seekingcertain data may be identified. The query may include aspects of datasuch as a schedule policy, storage policy, storage preference or otherpreference. The query may be analyzed and a primary metabase searched toidentify data objects that satisfy the query (block 1704). This mayinclude parsing the query into constituent parts and analyzing each partalone or in combination with other portions as part of the evaluationprocess. At block 1706, it may be determined, whether data objectssatisfying the query are to be copied to other primary storage devices,to secondary storage devices or both (pursuant to a storage policy orthe like).

If at least some data objects satisfying the search criteria are to becopied to other primary storage devices, those data objects may beidentified as further described herein and the target primary storagedevice(s) identified. This may involve consulting a storage policy orstorage manager to determine the destination point. In some embodiments,destination maybe determined dynamically, such that it is selected basedon certain system preferences or optimization routines that select astorage device based on storage capacity, availability, data paths tothe destination, combinations of the same or the like.

At block 1708 the identified data objects may be copied from primarystorage of a first computing device (the source) to primary storage of asecond computing device (the target or destination). Any metadataassociated with the first computing device describing the copied datamay also be copied to a metabase associated with the second computingdevice such that this description information is not abandoned or lost,but rather travels with the copied data for subsequent use (block 1710).

Next, at block 1712, it may be determined whether the copied dataobjects and associated metadata are to be deleted from the sourcecomputing device. For example, this may be done in order to free storagespace on the source computer or in accordance with certain data aging ormigration criteria. If it is decided to delete the data objects (andassociated metadata) the memory locations which include the data may beerased or designated for overwrite (block 1714 and 1716).

In some embodiments the data objects may be deleted but certain metadatamay be retained. If it is decided not delete the data objects, the datais retained and an index in an associated storage manager may be updated(block 1718), for example by updating an index to reflect a newlocation, data object status, any changes, combinations of the same orthe like, and return to block 1702. In other embodiments, if data isdeleted from the system, for example, a user permanently deletes certaindata from an application, that associated data may also be deleted fromboth primary and secondary storage devices and associated metabases tofree storage space within the system.

Returning to block 1706, it is also determined whether certain dataobjects currently stored in primary storage are to be migrated to one ormore secondary storage devices. If so, an archive file similar to theone described in FIG. 16 or other data structure suitable for transportmay be constructed or created by the source computing device withidentified data objects placed in the payload section and header andindex information added (block 1722). Data may be moved from primary tosecondary storage in predefined chunks which are constructed from sucharchive files, for example, using a data pipe, such as the data pipedescribed in U.S. Pat. No. 6,418,478, entitled “Pipelined High SpeedData Transfer Mechanism.”

Next, at block 1724 one or more target secondary storage devices may beidentified. This may involve consulting a storage policy or storagemanager to determine the destination point. In some embodiments,destination maybe determined dynamically, such that it is selected basedon certain system preferences or optimization routines that select astorage device based on storage capacity, availability, data paths tothe destination, or the like. Once the secondary storage device(s) areidentified, the archive files may be routed to a media agent, storagemanager, or other system component, that supervises the transfer to thetarget secondary storage device (blocks 1724 and 1728). This may involveselecting and appropriate data transfer route and ensuring the properresources and are available (e.g., bandwidth) such that the data may becopied with a certain period of time. Supervision may further includeparsing a copy operation into several portions with each portion beingtransferred by certain media agent or other resources, to meet system ortransfer requirements (e.g., a time window).

Next, the appropriate media within the target storage device may beidentified (block 1730) and the archive files may be transferred fromthe media management device to the secondary storage device (block1732). Such media may be selected from available media alreadyassociated with a similar data transfer or may be selected and reservedfrom an available media pool or scratch pool within the storage device.During or after the transfer, a media agent index or storage managerindex associated with the secondary storage device may be updated toreflect the transfer (block 1733). This may include copying theappropriate management files to the media management index such asoffset, media ID file name or other management information.

At block 1734, any metadata stored in a first metabase associated withthe transferred data objects may also be transferred and used to updatea second metabase associated with the target secondary storage device.Such metadata may be copied from the first metabase to the secondmetabase using network transmission resources. In some embodiments, themetadata in the first metabase may be deleted after it is confirmed themetadata has been copied to the second metabase. In other embodiments,the metadata may remain in both first and second metabases.

At block 1736, it may be determined whether the data objects transferredfrom the primary storage device are to be deleted. If so, the dataobjects and associated metadata in a first metabase may be erased orotherwise designated for overwrite (blocks 1738 and 1740). In somecases, a pointer or other reference such as a file stub may be left inthe original data location. In one embodiment, SM index is updated inblock 1742.

FIG. 18 presents a generalized block diagram of a hierarchicallyorganized group of storage operation cells in a system to performstorage operations on electronic data in a computer network inaccordance with an embodiment of the present disclosure. It will beunderstood that although the storage operation cells generally depictedin FIG. 18 have different reference numbers than the storage operationcell 1550 shown in FIG. 15, these cells may be configured the same as orsimilar to the storage cell 1550 as depicted in FIG. 15.

As shown, the system illustrated in FIG. 18 may include a master storagemanager component 1835 and various other storage operations cells. Asshown, the illustrative embodiment in FIG. 18 includes a first storageoperation cell 1840, a second storage operation cell 1845, a thirdstorage operation cell 1850, a fourth storage operation cell 1855, andmay be extended to include nth storage operation cell, if desired (notshown). However, it will be understood this illustration is onlyexemplary and that fewer or more storage operation cells may be presentor interconnected differently if desired.

Storage operation cells, such as the ones shown in FIG. 18 may becommunicatively coupled and hierarchically organized. For example, amaster storage manager 1835 may be associated with, communicate with,and direct storage operations for a first storage operation cell 1840, asecond storage operation cell 1845, a third storage operation cell 1850,and fourth storage operation cell 1855. In some embodiments, the masterstorage manager 1835 may not be part of any particular storage operationcell. In other embodiments (not shown), master storage manager 1835 mayitself be part of a certain storage operation cell. This logicalorganization provides a framework in which data objects, metadata andother management data may be hierarchically organized and associatedwith appropriate devices components (e.g., storage devices).

The storage operation cells may be configured in any suitable fashion,including those which involve distributed or centralized metabases. Forexample, storage operation cell 1840 may include a centralized primarystorage metabase and a centralized secondary storage metabase, storageoperation cell 1845 may include a centralized primary storage metabaseand multiple secondary storage metabases, storage operation cell 1850may include multiple primary storage metabases and a centralizedsecondary storage metabase, and storage operation cell 1855 may includemultiple primary storage metabases and multiple secondary storagemetabases (not shown). However, it will be understood that this ismerely illustrative, and any other suitable configuration may be used ifdesired.

Thus, in operation, master storage manager 1835 may communicate with amanagement agent of the storage manager of the first storage operationcell 1840 (or directly with the other components of first cell 1840)with respect to storage operations performed in the first storageoperation cell 1840. For example, in some embodiments, master storagemanager 1835 may instruct the first storage operation cell 1840 withcertain commands regarding a desired storage operation such as how andwhen to perform particular storage operations including the type ofoperation and the data on which to perform the operation.

Moreover, metabases associated with each storage operation cell maycontain information relating to data and storage operations as describedherein. In some embodiments, master storage manager 1835 may include amaster metabase index or database (not shown) that reflects some or allof the metadata information from the hierarchically subordinate storageoperation cells within the system. This allows the system to consult themaster storage index or database for information relating to data withinthose storage operation cells rather than requiring each cell becontacted of polled directly for such information.

In other embodiments, master storage manager 1835 may track the statusof its associated storage operation cells, such as the status of jobs,system components, system resources, and other items, by communicatingwith manager agents (or other components) in the respective storageoperation cells. Moreover, master storage manager 1835 may track thestatus of its associated storage operation cells by receiving periodicstatus updates from the manager agents (or other components) in therespective cells regarding jobs, system components, system resources,and other items. For example, master storage manager 1835 may usemethods to monitor network resources such as mapping network pathwaysand topologies to, among other things, physically monitor storageoperations and suggest, for example, alternate routes for storing dataas further described herein.

In some embodiments, master storage manager 1835 may store statusinformation and other information regarding its associated storageoperation cells and other system information in an index cache, databaseor other data structure accessible to manager 1835. A presentationinterface included in certain embodiments of master storage manager 1835may access this information and present it to users and system processeswith information regarding the status of storage operations, storageoperation cells, system components, and other information of the system.

In some embodiments, master storage manager 1835 may store and/or trackmetadata and other information regarding its associated storageoperation cells and other system information in an index cache, databaseor other data structure accessible to manager 1835. Thus, during asearch procedure as further described herein, queries can be directed toa specific storage operation cell or cells based on the cellos function,past involvement, routing or other information maintained within thestorage manager or other management component.

As mentioned above, storage operation cells may be organizedhierarchically. With this configuration, storage operation cells mayinherit properties from hierarchically superior storage operation cellsor be controlled by other storage operation cells in the hierarchy(automatically or otherwise). Thus, in the embodiment shown in FIG. 18,storage operation cell 1845 may control or is otherwise hierarchicallysuperior to storage operation cells 1850 and 1855. Similarly, storageoperation cell 1850 may control storage operation cells 1855.Alternatively, in some embodiments, storage operation cells may inheritor otherwise be associated with storage policies, storage preferences,storage metrics, or other properties or characteristics according totheir relative position in a hierarchy of storage operation cells.

Storage operation cells may also be organized hierarchically accordingto function, geography, architectural considerations, or other factorsuseful or desirable in performing storage operations. For example, inone embodiment, storage operation cell 1840 may be directed to createsnapshot copies of primary copy data, storage operation cell 1845 may bedirected to create backup copies of primary copy data or other data.Storage operation cell 1840 may represent a geographic segment of anenterprise, such as a Chicago office, and storage operation cell 1845may represents a different geographic segment, such as a New Yorkoffice. In this example, the second storage operation cells 1845, 1850and 1855 may represent departments within the New York office.Alternatively, these storage operation cells could be further divided byfunction performing various types of copies for the New York office orload balancing storage operations for the New York office.

As another example, and as previously described herein, differentstorage operation cells directed to different functions may also containthe same or a subset of the same set of physical devices. Thus, onestorage operation cell in accordance with the principles of the presentdisclosure may be configured to perform SRM operations and may containthe same, similar or a subset of the same physical devices as a cellconfigured to perform HSM or other types of storage operations. Eachstorage operation cell may, however, share the same parent or,alternatively, may be located on different branches of a storageoperation cell hierarchy tree. For example, storage operation cell 1845may be directed to SRM operations whereas storage operation cell 1855may be directed to HSM operations. Those skilled in the art willrecognize that a wide variety of such combinations and arrangements ofstorage operation cells are possible to address a broad range ofdifferent aspects of performing storage operations in a hierarchy ofstorage operation cells.

In some embodiments, hierarchical organization of storage operationcells facilitates, among other things, system security and otherconsiderations. For example, in some embodiments, only authorized usersmay be allowed to access or control certain storage operation cells. Forexample, a network administrator for an enterprise may have access tomany or all storage operation cells including master storage manager1835. But a network administrator for only the New York office,according to a previous example, may only have access to storageoperation cells 1845-1855, which form the New York office storagemanagement system.

Moreover, queries performed by the system may be subject to similarrestrictions. For example, depending on access privileges, users may belimited or otherwise excluded from searching a certain cell or cells.For example, a user may be limited to searching information in cells ormetabases within the system that are unrestricted or to those whichspecific access rights have been granted. For example, certain users maynot have privileges to all information within the system. Accordingly,in some embodiments, as a default setting, users may have accessprivileges to information in cells that they interact with. Thus,confidential and sensitive information may be selectively restrictedexcept only to certain users with express privileges (e.g., financial orlegal information). For example, certain classification informationwithin the metabases in the system may be restricted and thereforeaccessed only by those with the proper privileges.

Other restrictions on search criteria may include the scope of thesearch. For example, in a large network with many storage cells mayrequire dedicating significant amounts of resources to perform go globalor comprehensive searches. Thus, if a certain resource threshold isexceeded by a proposed search, the system may prompt that search to bemodified or otherwise cancelled.

In other embodiments master storage manager 1835 may alert a user suchas a system administrator when a particular resource is unavailable orcongested. For example, a particular storage device might be full orrequire additional media. For example, a master storage manager may useinformation from an HSM storage operation cell and an SRM storageoperation cell to present indicia or otherwise alert a user or otherwiseidentify aspects of storage associated with the storage managementsystem and hierarchy of storage operation cells.

Alternatively, a storage manager in a particular storage operation cellmay be unavailable due to hardware failure, software problems, or otherreasons. In some embodiments, master storage manager 1835 (or anotherstorage manager within the hierarchy of storage operation cells) mayutilize the global data regarding its associated storage operation cellsto suggest solutions to such problems when they occur (or act as awarning prior to occurrence). For example, master storage manager 1835may alert the user that a storage device in a particular storageoperation cell is full or otherwise congested, and then suggest, basedon job and data storage information contained in its database, orassociated metabase, or an alternate storage device. Other types ofcorrective actions based an such information may include suggesting analternate data path to a particular storage device, or dividing data tobe stored among various available storage devices as a load balancingmeasure or to otherwise optimize storage or retrieval time. In someembodiments, such suggestions or corrective actions may be performedautomatically, if desired. This may include automatically monitoring therelative health or status of various storage operation cells andsearching for information within the cells of the system relating tosystems or resource performance within that cell (e.g., index, metabase,database, or the like) for use in diagnostics or for suggestingcorrective action.

In certain embodiments, HSM and SRM components may be aware of eachother due to a common database or metabase of information that mayinclude normalized data from a plurality of cells. Therefore, in thoseembodiments there is no need for such information to pass through amaster storage manager as these components may be able to communicatedirectly with one another. For example, storage operation cell 1845 maycommunicate directly with storage operation cell 1855 and vice versa.This may be accomplished through a direct communications link betweenthe two or by passing data through intermediate cells.

Moreover, in some embodiments searches may be performed across anumerous storage cells within the hierarchy. For example, a query may beposed to master storage manager 1835 that may pass the query downthrough the hierarchy from cells 1840 to 1845 to 1850 and 1855. This maybe accomplished by passing the query form one manager component of eachcell to another, or from one data classification agent to another, onemetabase to another. The results may be passed upward through thehierarchy and compiled with other results such that master storagemanager 1835 has a complete set of results to report. In otherembodiments, each storage manager cell may report results directly tothe requestor or to a designated location.

FIG. 19 presents a flow chart 1900 that illustrates some of the blocksthat may be involved in performing searches for data objects acrosssystems that include multiple primary and secondary storage devices,according to certain embodiments of the present disclosure. First, atblock 1905, a query seeking certain data may be identified (e.g., from astorage policy, user preference, other process, or the like). The querymay be analyzed to identify system components, such as clientspotentially having information such as certain data objects or metadatathat may satisfy the query (e.g., by excluding certain clients that areunlikely to have data being sought based on certain query parameterssuch as location, time frame, client or other component, department,application type, or any other criteria used to classify data asdescribed herein, combinations of the same or the like (block 1910)).Results may be presented based on a confidence factor indicating thelikelihood that the results meet the specified parameters. For example,results substantially satisfying most or all criteria may be listedfirst with the confidence factors provided based on a percentage of thecriteria satisfied (e.g., a query that returned results having three outof four criteria satisfied may be represented with a 75% confidencefactor or the like). Less relevant results may be listed subsequentlywith the confidence factor provided based on any suitable relevantfactor such as number of parameters satisfied, how close the match is,combinations of the same or the like.

The search process may further involve consulting one or more indexesassociated with the clients to identify where responsive data objects orother copies of client data, may be located within the system. At block1915, it may be determined whether client data objects satisfying thequery are located in primary storage, secondary storage, or both (e.g.,based on index information in a storage manager). This may be based onpolling various storage manager or a master storage manager thatincludes information the covers or represents whole system or theportion of system specified for search.

If it is determined that responsive data objects are only located onclient(s) in primary storage, that client may be added to the list ofclients to be searched (block 1950). If it is determined that responsivedata objects are located in secondary storage devices (or other primarystorage locations that may be identified), the system may consult astorage manager index to identify archive files (or other files) basedon certain query parameters such as a specified point in time,origination point, or the like, or on index data stored in a storagemanager index identifying archive files or other file associated withthe data objects.

Next at block 1920, storage managers may be consulted to identifyresponsive archive files. At block 1925, media management componentsthat may have handled responsive data objects are be identified. Thismay be based on information retrieved from the storage manager indexregarding archive files, e.g., an association of archive files withmedia agents and media items. It may then be determined whether theidentified media management components (“MMCs”) have metadata relatingto the identified archive files available readily available in an indexcache (block 1930).

This may be accomplished by searching for reference information relatingto the identified archive files. If such information is already presentin the cache, responsive data objects may be identified and retrievedusing the index cache information, which may include, offsets and anyfile identifiers or the like, by the media management component, and thesystem may proceed to block 1940 (determine whether another mediamanagement component needs to be analyzed).

If not, the index information may need to be loaded from the secondarystorage device so archive files may be retrieved and accessed. This mayinvolve identifying the particular media on which the index data isstored and upload it to the media management component cache (block1935). In some embodiments, a master storage manager or other componentwith information relating to files may be consulted to identify mediacontaining the responsive information. These media may be mounted indrive or other input/output device and examined to locate the properfiles or data structures. Index information may then be located anduploaded to an index or database associated with the appropriate mediamanagement component (e.g., media agent). This allows the mediamanagement component to locate and retrieve specific data objects on themedia that satisfy the search criteria.

Next, if no further media management components have been identified, alist of media management components to be searched may be compiled(block 1945). At block 1950, a list of clients identified as potentiallyhaving responsive data objects may also be compiled. After a completelist of secondary storages devices and clients potentially havingresponsive data objects is identified (via block 1955), the associatedmetabases are queried for these components, in block 1960, and resultsare returned indicating data objects that may satisfy the searchcriteria, in block 1965. In some embodiments, these results may bereviewed and analyzed to ensure relevance, with only reasonably relevantor responsive data objects actually being retrieved.

FIG. 20 presents a flow chart 2000 that illustrates some of the blocksthat may be involved in retrieving data objects from secondary storage(or other tiers or other storage locations) in accordance withprinciples of the present disclosure. This may be accomplished generallyas follows. Certain data (e.g. data objects or associated metadata) fromthe system may need to be retrieved. That data may be requested andcommunicated to the system in the form of a query. The query may be usedto search the system and identify media on which responsive data may belocated. Once located, data satisfying the selection criteria may beuploaded and retrieved and analyzed for relevance, or other action maybe taken. Or, alternatively, the identified data may be moved to othertiers of storage. More specific blocks involved in this process may beas follows.

First, at block 2002, a query seeking certain data may be identified.The query may be analyzed to ascertain certain additional informationthat may assist in identifying responsive information such asidentifying a certain point in time to search (block 2004). This mayinvolve consulting storage manager and/or media agent index or databasefor responsive information relating to a certain point in time. This mayalso involve consulting certain metabases for similar information thatmay be associated with these or other media management componentsproviding copy and management functions. Point in time information maybe specified by the user or may be assigned by the system absent aspecific time frame established by the user. For example, a user mayspecify a certain time range within the query (e.g., a time range, acertain date, all information related to a project since its inception,combinations of the same or the like). The system however, may assign acertain time limit based on the query (e.g., such as based on thespecifics of the query (e.g., only have data relating to a certain timeframe)), and may limit the search to the time frame of informationpresent in certain metabases, master storage manager, or index withinthe system, and/or poll or otherwise communicate with storage deviceswithin the system to determine the range or time frame of available datawithin the system and present the user with options for retrieving it(e.g., some or all within a time frame)

Next, at block 2006 certain archive files may be identified andassociated media agents (block 2008) that may have been involved intransferring responsive data objects. This may be determined byconsulting a master storage manager or other media management componentindex or metabase to determine whether the archive files have beenhandled by such components. Once the appropriate media agents have beenidentified, it may be determined whether information regarding theidentified archive files is present in a cache or index associated withthe media agents (block 2010). If not, the index information may need tobe uploaded so the appropriate archive files may be retrieved andaccessed (block 2012). This process may be performed (via block 2014)until all identified media agents have the appropriate index informationloaded and/or until it is determined that no responsive information hasbeen handled by the media agents and therefore no index information needbe uploaded.

Next, at block 2016 data objects satisfying the query criteria may beidentified by searching metabases and/or indexes. In some embodiments,such data objects may be compiled into a list of data objects forpresent or subsequent retrieval. For example, such a list of responsivedata objects may be provided to the user (which may itself satisfy thequery) and then provide the user with the option to actually retrieveall or certain selected identified data objects.

At block 2018, the new destination for the data objects may bedetermined. For example, if certain data objects are being migrated offas part of an ILM operation, the query or other information may indicatethe intent or reason for the search and the data object's destination.This may be useful in determining whether certain data objects areresponsive to search criteria or query. At block 2020 it may bedetermined whether the new destination is primary storage (a restoreoperation) or secondary or other tier of storage (ILM). Such informationmay be further useful in determining whether the data objects are likelyto fall within a time frame or category of interest and thus may beuseful in further identifying data objects of interest.

If the identified data objects are moving to other secondary storagetiers, the data objects may be repackaged into form suitable forsecondary storage, which may include repackaging into an archive file,converting to a new format, compressing of the data objects andassociated files, encryption, or any other containerization techniqueknown in the art (block 2022).

Once the data objects are in a suitable format, they may be copied tothe appropriate storage destination by the system. This may beaccomplished by a media agent or media component in conjunction with astorage manager or other media management component that coordinaterouting and the specifics involved with file transfer (block 2024), asfurther described herein. Metadata relating to the copied data objectsmay then be copied to a metabase associated with a computing device atthe destination (block 2026).

For example, metadata relating to the data being copied may be copiedalong with the data to the secondary storage device and may be copied toan index in the media agent or other media management component involvedin the data transfer. This allows the media management component tolocate and retrieve and otherwise manage the stored data. Such metadatamay also useful when performing searches of secondary storage devices(or other tiers) as further described herein. Metadata stored along withthe data on the secondary storage device may be useful to restore orrefresh the media agent index in the case of lost or corrupt data andalso may be transferred along with the data on storage media in the casewhether it is necessary to copy all such data (or actually physicallyrelocate) to another storage device. A master storage manager index ormetabase associated with destination computing device may be updatedreflecting the arrival and new location of the transferred data objectsand/or archive file for system management purposes (block 2034).

In some embodiments, the copied data objects and metadata may be deletedfrom the source location (blocks 2028-2032). For example, at block 2028,it may be determined whether the copied data objects should be deletedbased user preferences, storage policy requirements or other systemconstraints such has diminished storage capacity. At blocks 2030 and2032 the data objects and records may be deleted. However, a stub,pointer or other referential element may be placed at the same logicallocation to act as a marker for the moved data. This allows subsequentoperations to quickly track down and locate the moved data at its newlocation.

If, however, at block 2020, it may be determined that the identifieddata objects are moving to primary storage, accordingly, the dataobjects may be reformatted (e.g., unpacked from archive file format) forcopying to a computing device (block 2038). Next the unpacked data maybe copied to a target computing device along with any associatedmetadata (blocks 2040 and 2042). For example, this may involve readingmetadata and/or index information from the archive file and repopulatingthe metabase and/or management component indexes with this informationas further described herein. For example, metadata from the archive filemay be retrieved and integrated into a metabase associated with thetarget computing device including information relating to datamanagement and as well as certain content and storage information asfurther described herein with respect to the classification process andmetabase population. Thus, such archive information may be fullyrestored to primary storage and any associated information, such asmetabase information may be searched and retrieved accordingly.

Moreover, information relating to system management may be uploaded andused to repopulate storage management components within the system suchas a storage manager or master storage manager reflecting the return ofthe retrieved data to primary storage (block 2050). For example, astorage manager index may be updated to reflect the presence of theretrieved data along with certain management information such as logicaloffsets and location of the retrieved information such that theretrieved information may be located and accessed. Other managementcomponents, such as a master storage manager may also be updated withthe appropriate identification and location information to reflect thereturn of the retrieved data within the system.

In certain embodiments, the copied data and metadata may be deleted fromthe source location (blocks 2044-2048). For example, at block 2044, itmay be determined whether the copied data objects in secondary storageshould be deleted based user preferences, storage policy requirements orother system constraints such has diminished storage capacity. At blocks2046 and 2048 the data objects and records may be deleted within thesystem including any metabase or other system management informationassociated with the retrieved data. Storage management components suchas storage managers, media agents may also be updated to reflect theremoval or deletion of such information (block 2050).

Systems and modules described herein may comprise software, firmware,hardware, or any combination(s) of software, firmware, or hardwaresuitable for the purposes described herein. Software and other modulesmay reside on servers, workstations, personal computers, computerizedtablets, PDAs, and other devices suitable for the purposes describedherein. Software and other modules may be accessible via local memory,via a network, via a browser or other application in an ASP context, orvia other means suitable for the purposes described herein. Datastructures described herein may comprise computer files, variables,programming arrays, programming structures, or any electronicinformation storage schemes or methods, or any combinations thereof,suitable for the purposes described herein. User interface elementsdescribed herein may comprise elements from graphical user interfaces,command line interfaces, and other interfaces suitable for the purposesdescribed herein. Screenshots presented and described herein can bedisplayed differently as known in the art to input, access, change,manipulate, modify, alter, and work with information.

While the invention has been described and illustrated in connectionwith preferred embodiments, many variations and modifications as will beevident to those skilled in this art may be made without departing fromthe spirit and scope of the invention, and the invention is thus not tobe limited to the precise details of methodology or construction setforth above as such variations and modification are intended to beincluded within the scope of the disclosure.

1. A method for managing the transfer of electronic data in a storagesystem, the method comprising: receiving a monitor index having aplurality of entries identifying data interactions made by at least oneapplication to a plurality of data objects on first and second storagedevices, the entries in the monitor index being populated by ajournaling application; updating, based on said monitor index entries,metadata stored in a metabase; evaluating a selected criteria fortransferring at least one data object from the first storage device tothe second storage device, said evaluating comprising accessing themetadata stored in the metabase; and identifying a first data object, ofthe plurality of data objects, that satisfies the selected criteria. 2.The method of claim 1, further comprising performing at least onetransferring operation on the first data object based on the selectedcriteria.
 3. The method of claim 1, wherein said accessing of themetadata stored in the metabase is performed without accessing the dataobjects stored in the first storage device.
 4. The method of claim 1,further comprising accessing one or more metadata associated with thecorresponding data objects again one or more times to update themetabase.
 5. The method of claim 1, wherein said identifying furthercomprises providing a file reference number associated with the firstdata object.
 6. The method of claim 1, wherein said identifying furthercomprises providing a path indicative of a location of the first dataobject on the first storage device.
 7. The method of claim 1, whereinthe selected criteria comprises at least one of a schedule policy, astorage policy and a migration policy.
 8. The method of claim 1,additionally comprising copying the first data object from a firstlocation on the first storage device to a second location on the secondstorage device.
 9. The method of claim 8, additionally comprisingupdating the metabase to reflect the second location of the first dataobject on the second storage device.
 10. The method of claim 9, whereinthe metabase comprises first and second metabases, wherein the firstmetabase is associated with the first storage device and the secondmetabase is associated with the second storage device.
 11. The method ofclaim 10, wherein said updating comprises updating one or more recordsof the second metabase.
 12. The method of claim 11, additionallycomprising removing records on the first metabase associated with themetadata of the first data object.
 13. The method of claim 8,additionally comprising storing on the first storage device referencedata that identifies the second location of the first data object on thesecond storage device.
 14. A system for managing the transfer ofelectronic data between a plurality of storage devices, the systemcomprising: a first storage device having a plurality of data objects; asecond storage device; a selected criteria for copying at least one ofthe plurality of data objects on the first storage device to the secondstorage device; at least one metabase having a plurality of recordscomprising metadata associated with the plurality of data objects; aclassification module configured to evaluate a monitor index generatedby a journal program, the monitor index comprising a list ofinteractions between at least one application and the plurality of dataobjects, the classification module being further configured to updatethe metadata stored in the at least one metabase based on interactionsidentified by the monitor index; and a management module configured toaccess the at least one metabase to identify a first data object, of theplurality of data objects, that satisfies said selected criteria. 15.The system of claim 14, wherein the interactions identified by themonitor index comprise interactions that modify metadata of at least aportion of the plurality of data objects.
 16. The system of claim 14,wherein the management module accesses the at least one metabase withoutaccessing the plurality of data objects on the first storage device. 17.The system of claim 14, wherein the journal program is different thanthe at least one application.
 18. The system of claim 14, wherein theclassification module is located outside an input/output (I/O) pathbetween the at least one application and the first storage device. 19.The system of claim 14, wherein the classification module is managed bya first processor and the interactions are managed by a secondprocessor.
 20. The system of claim 14, wherein the management module isfurther configured to transfer the first data object from the firststorage device to the second storage device.
 21. The system of claim 20,wherein the first storage device comprises a faster access time than thesecond storage device.
 22. The system of claim 14, wherein the at leastone metabase comprises a first metabase and a second metabase, whereinrecords stored in the first metabase correspond to metadata of theplurality of data objects stored on the first storage device, andrecords stored in the second metabase correspond to metadata of a secondplurality of data objects stored on the second storage device.
 23. Thesystem of claim 22, further comprising a master index identifyinginformation stored on the first and second metabases.
 24. The system ofclaim 14, wherein at least a portion of the at least one metabase isstored on the first storage device.
 25. A system for managing thetransfer of electronic data stored on a plurality of storage devices,the system comprising: first means for storing a first plurality of dataobjects; second means for storing a second plurality of data objects;means for identifying at least one of the first plurality of dataobjects to be transferred from the first storing means to the secondstoring means; third means for storing metadata associated with thefirst plurality of data objects; and means for processing a journal filecomprising a list of interactions between at least one application andthe plurality of data objects, said processing means being furtherconfigured to update said third storing means based on interactionsidentified by the journal file; and means for accessing said thirdstoring means to identify a first data object, of the plurality of dataobjects, that satisfies said selected criteria.
 26. The system of claim25, wherein said metadata is stored separate from the entire contents ofthe data objects.
 27. The system of claim 25, wherein said identifiedinteractions comprise interactions that modify metadata of at least aportion of the plurality of data objects.
 28. The system of claim 25,wherein said accessing of said third storing means is performed withoutaccessing the plurality of data objects on the first storing means. 29.The system of claim 25, wherein said selected criteria comprises a datastorage migration policy.